Everyone who's even considering Caltech knows that Caltech's academics are incredibly challenging. The learning done at Caltech is indubitably one of the best in the nation. To get into this place, you'd have to be an incredible scholar. They wouldn't let you in otherwise.
However, the biggest problem with Caltech's academics is that our professors are very good at their research. They are so good with their research that they value their research above their teaching and above the people they teach. It's not like they hate people (for the most part), it's just that they're less of a priority for them. Techers don't learn quite as much from their professors as they do from figuring out the problem sets with each other. In some classes, most of the students taking that class won't even show up to lecture regularly. It's a running joke that professors hold lectures in lecture halls that can accommodate fewer people than the number of people in the class because by the end of the first week of term, already half of the people won't even show up.
Since Techers learn from each other, competing with each other is out of the question. Collaboration is essential for completing the otherwise-impossible problem sets. As far as core goes, Techers learn math from the phys majors, chemistry from the chem majors, and so on. Also, all the upperclassmen have taken the core classes, so the frosh can just find an upperclassman that remembers what to do (harder than you'd think) and ask how to do the set. Finally, the deans' office provide free tutors for everyone who would ask.
What about core? Core is five terms of physics (covering classical mechanics, relativity, electromagnetism, waves, quantum mechanics, and statistical mechanics), five of math (single and multivariable calculus, linear algebra, differential equations, probability, and statistics), two of chem, and one of bio. Core is a good idea before and after you go through it, but while you're going through it, it's not as pleasant. Most colleges go through quantum mechanics in at least a semester. Non-phys majors at Caltech go through quantum in six weeks. Granted, if you're in a major that actually needs quantum, they reteach it to you in even more gory detail, but Ph 002A is still pretty bad.
The honor code ("any member of the Caltech community shall not take unfair advantage of another") is amazing. Take home quizzes and finals are common. You can take your finals in your room and time yourself. You get access to buildings. You can even buy lunch and pay for it the next day if you want. People trust you with almost anything. Don't break that trust.