Thanks for your questions “How difficult is it to start a cafe, bakery, or restaurant business?” I think this questions restaurant or cafe is very important for all like yours.
Here I’m trying to give your answers in brief.

My wife is in restaurant and bakery business for a few years now. Several of my friends and cousins own restaurants in US and India.
Starting a cafe/bakery/restaurant (or any business for that matter) is not that difficult if you can put together the investment needed. Making it successful (“Execution” in business speak) is whole another matter. According to a hospitality industry research report, 60% of the restaurants fail or change hands within first 3 years.

Initial investment to start a full-service restaurant widely varies. It can be anywhere between $200,000 to $2,000,000 to build a restaurant based on how much you want to spend on interiors and kitchen equipment etc… Cafes and bakery can cost much less. somewhere between $50,000 to $300,000

Breakeven: I have seen some hospitality businesses break even within few months. I know some are still losing money after years of operation. First year is the very crucial for a hospitality business. That’s when reputations are made or lost. I think, a well located and well run restaurant should break even by the end of first year. A small fraction of restaurants make very good money. Vast majority of them lose money or barely break even.

Challenges

Oh boy! where do I start? There are lot of challenges in hospitality industry, some are unique to this industry.

Partners: I have seen restaurants fail because partners can’t work together. This happens mostly because of contribution asymmetry. Usually, the roles are not clear. Some partners can add value like a great chef or a good manager. But, others may not have time, energy or skill. Make sure that roles are clear and compensation for the effort is clearly defined and agreed upon. It always helps to have an outside trusted advisor.
Decision making process among partners: Lot of small businesses don’t have a well defined decision making process.
Location: Location is extremely important for any retail business. Customers have lots of options. I have seen mediocre restaurants survive because of location. E.g. some average Indian restaurants in small towns do well because it’s the only game in town. Some decent restaurants didn’t survive because there are 10 restaurants that serve similar food within two miles.
Spending too much money before restaurant starts: This happened to us. Hospitality business has high fixed costs like payroll, rent, utilities and other services. You need to have enough money for operating expenses because it’s very difficult to estimate when you break even.
Family life goes out of the window: Hospitality is very high touch business. you can’t leave it on employees. The owner needs to be present almost all the time at least until it establishes. You will be spending 70 hours a week on business. Keep in mind that you will be busy when your family wants to spend time with you the most. You will be extremely busy in the evenings and weekends. I don’t get to spend time with my wife on valentine’s day because that’s a busy day for restaurants. Our social life changed drastically because my wife is busier just when my family and friends want to spend time together 🙁
Your specialty: You got to have your own specialty. Your restaurant name should pop into customer’s mind when they are in mood for a certain type of food.
Customer service: You are not selling a household product. You are selling an experience. Experiences appeal to human emotions more strongly than physical products. This is most important for a hospitality business. In my opinion, this is more important than taste of food. Majority of customers want to try new restaurants. So, you always get an opportunity to prove yourself. But, you only get one opportunity to please them.
Cashflow: You need to be extremely careful with your cashflow. Your payroll is your biggest expense. More than inventory and rent. Keep it on leash.
Your brand and marketing: Brand is everything for hospitality business. You need to create positive buzz. The positive buzz takes on a life of its own.
People management: Hospitality business has high employee turnover. It takes a lot of time to train new employees. You need to have a very cool head. Imagine a scenario where all the following are happening at the same time:
Restaurant is busy in the evening
Everyone is tired
a customer is unhappy and creating a scene
5 other tables are waiting for food
Food runner is putting pressure on the chef
Some equipment stopped working
Tempers are rising
Chef is screaming on line cook
You just happened to see a bad review of yelp for no fault of yours
Dishwasher just quit on you
Stuff like this happens more often than you think.

The most important challenge is “Hospitality business is not right for you”: I know lot of people that regretted getting into this business. Hospitality business appeals to a lot of people because it’s such a common personal experience. We all cook at home. We know how much it costs. We pay a lot more at the restaurant for food that sometimes may not appeal to our taste buds. It’s very easy for the customers to notice what is wrong with a restaurant and easy to come to a conclusion that they can do a better job. Hospitality business looks very glamourous as well. This business is not right for you, unless you are very passionate about this business, You have something valuable to contribute, don’t mind working 70 hours a week, don’t mind not having work life balance and don’t mind not making lot of money.

Thanks for your questions. If you find this answer is useful for you then share with your friends on facebook, twitter and google plus. If you do not get your answer clearly then ask me again below or comment on my answers.
If you have any questions, please feel free to ask me here.