Blue,

I don't think your crazy at all.

My typical summer base pack

 weight ( all my stuff sans food, fuel, water ) for backpacking is 4.8 lbs.

Bike touring tends to be heavier because your carry bike tools, may need more privacy or weather protection than a poncho tarp offers if you stay at public campgrounds and can't be as choosy about where you pitch your shelter, and may require a thicker sleeping pad ( Which you didn't mention ) since most public campgrounds tend to have hard surfaces to camp on instead of deep forest duff.

I wouldn't worry about the cost of your bike.

Some of the most popular lugged steel touring bikes are quit expensive..over $3000.00 for a Rivendell Atlantis. My own '91 Bridgestone RB-T may even prove to be irreplaceable.

Just take a good lock, maybe replace your wheel QR with a regular nut if wheel theft worries you. There are gadgets on the market for wheels, seatpost, et al.

Be careful where you put your bike. Place it where it can be seen by non-thieves or where you can easily see it from the store you are in. Check on it frequently. A system that allows you to carry your valuables with you,,like a  CamelBak pack, is a good idea. 

You aren't crazy to not use panniers with an ultralight load.

I use a 21 liter saddle bag and a Wald medium basket up front and that's all I need for a week of un-resupplied touring....many do well with less and lighter than I.

A high center of gravity is not the big issue it becomes with conventional touring loads so panniers aren't necessary and may even cause handling problems if you have to mount them far back of the axle for heel clearance ,which you may be lacking on a short chain stay road bike.

 One difference I have from you is I use a solo tarp-tent that weighs in the neighborhood of 1.5 lbs.

Either a Henry Shires Contrail or a Six Moon Designs Lunar solo.

Single wall...but well ventilated  and more privacy in a public CG than a tarp.

 Thermarest  has  a new mattress this spring called the NeoAir that is 2.5" thick, very light for that much thickness, around 3/4 lb. in a 20" x  72" or 20" x 66"....9 oz. in a 20" x  47" and packs down to less than the size of a 1 liter water bottle. Gravel or hard packed tent site..no problem.

Expensive though. 

I'd say your set to go.

Have a wonderful trip!!

GND