The Raff family hailed from the village of Wiesenstetten, in the south west German Duchy of Württemberg. The nearest town was Empfingen in the Black Forest, itself not far from Stuttgart, Württemberg's capital. Raff's paternal grandfather was Anton Raff - a respected, but not wealthy, weaver and farmer in the village. Anton and his wife Helene Lohmiller raised nine children, of whom the first was Michael. Raff's father, Franz Josef, followed in 1789. He was an intelligent, musically-inclined child whose eagerness to learn was encourage by his father. During the Napoleonic wars, Württemberg was elevated to a kingdom in 1806 and in 1809 joined France in its war on Prussia, Austria and Russia. More than 16,000 young men over 17 were conscripted into Napoleon's army, many forcibly, and this was Michael's fate. He was never heard of again by the family, and no doubt died during one of Napoleon's campaigns in the east. To avoid the same fate Franz Josef, with the help of his father, absconded and fled to neighbouring Switzerland, where he was taken in by the monks at a Cistercian Monastery in Wettingen, who not only fed him, but carried on his education until he was able to repay their generosity by himself teaching the younger boys at the monastery's school. He stayed until Napoleon's downfall in 1814/15 and left the Abbey once an amnesty for draft-dodgers was announced. He became the private tutor to the children of a wealthy family in Lucerne, but soon then gained the post of teacher at the school in Lachen, on the shores of Lake Zurich. Although Raff's father had eight siblings only one another brother is recorded. This is Mätthaus Raff, four years his junior, who entered the priesthood in 1828 and became parish priest of Oberkirchberg near Wiblingen, although he later gained a more senior clerical position as a dean associated with Ellwangen Abbey. Raff stayed with his uncle in the mid-1830s whilst studying at the Gymnasium in Rottweil, Württemberg, but Mätthaus strongly disapproved of Raff's later decision to become a musician.
There is little information about the Raff family who Franz Joseph had left behind in Wiesenstetten. An old LP booklet shows a 1930 photo of the Raff family house in Wiesenstetten, which was demolished in 1936. In front of is standing a Joachim Raff, described as a " descendent" of the composer who died in WWII. As Raff's only child had no children herself, this "descendent" Joachim is therefore more likely a descendent of Raff's uncles or of his brother Peter. Also in the picture is an older man, Anton Raff, who is described as a farmer and a great-nephew of the composer. He was presumably a son of Peter as Caspar, about whose life more is known, had emigrated to the USA and his children all grew up there.
In Lachen in 1819 Franz Josef Raff married a local woman who had previously planned to become a nun - 19 year old Katharina Schmid. Her father was Franz Joachim Schmid, who was and important citizen, not just of Lachen, but of the whole canton of Schwyz. A cattle farmer, he was at the time a cantonal Landammann - chairman of the local assembly, and continued to hold various prominent political and judicial positions into the 1830s. In 1831 he was involved in the secession from Schwyz of a number of more progressive northern districts which created the half-canton of Ausserschwyz (Outer Schwyz), of which Schmid was elected Landammann. On reunification of Schwyz in 1833, he transferred his political allegiance to the conservatives. He was elected to the cantonal council and in 1836 became Kantonsstatthalter, the cantonal governor, but was removed from the post two years later having become a leading member of the successful traditionalist side in the bitter "Horn and Hoof" dispute. The bitterness generated in Lachen by the dispute forced Franz Joseph and Katharina Raff to leave, and take their growing family first to nearby Schmerikon and then to the cantonal capital, Schwyz, where Franz Joseph was able to secure a good position as teacher at the town's Jesuit College, where they remained until his retirement on 1851. In that year the family left Switzerland and settled in the south Württemberg town of Ravensburg. At around that time they met Hans von Bülow, and he described the meeting to his friend Raff, who had been virtually disowned by his parents after his decision in 1844 to give up teaching for music. Bülow wrote that "despite his 62 years of age, your father looks very vigorous; I liked the clever, intelligent eyes and his great liveliness", but that he was "under the regime of your mother and eldest sister". He reported "the extremely narrow-minded views of your mother, who has lost all sympathy and love for you". Raff's mother showed no interest in Bülow's news of him, but "your father [...] took an intimate interest in you, his love for you was never completely denied. [...] He accompanied me out to the steps alone, and shook my hand several times; there were bright tears in his eyes when he said warm greetings to you [...] this paternal love, held back in the narrow room, which was openly expressed here in the open air, touched me: perhaps especially the contrast that it made with the heartlessness of the others." Both Franz Joseph and Katharina continued to live in Ravensburg until their deaths in 1861 and 1875.
Joseph and Katharina Raff's first child was Franz Joseph, but he died in infancy and so Joachim, born 14 months after Franz on 22 May 1822 became the eldest of their surviving children. The couple went on to have a further eight children over the next 15 years. Helene Raff mentions that their third child, Catharina, died in childhood and by the time the fourth, Maria Antonia, was born Raff was seven years old and the subject of a strict educational regime enforced by his father, so she was too young to replace Catharina as a playmate for him. As the family continued to grow Raff was sent off to school in Württemburg and this no doubt further contributed to the distance which established itself between him and his younger siblings. Helene Raff mentions only in a footnote Raff having six siblings: the deceased Catharina and Maria Antonia, Caspar, Aloysia, Seline and Peter. Hans von Bülow lists only those five children in his report of his meeting with the family in 1851, from which it may be assumed that the two sisters Maria Anna Diethelmina and Maria Anna Henrica had also died by then, probably in infancy. The baptismal names of Aloysia and Peter were Magdalena Christina and Joseph Michael, but they don't seem to have been used by the family. Each of the three Raff brothers was christened with Joseph as his first name, followed by their more familiar one. All the children seem have accompanied their parents in the move from Switzerland to Württemburg when their father retired in 1851. With the exception of the middle son Caspar, nothing is known about the lives of these younger children apart from the briefest of mentions in Helene Raff's biography and in von Bülow's 1851 letter, which gives an unflattering portrait of the eldest daughter, 22 year-old Maria Antonia: "who by the way does not look like you [Raff] at all, but looks very old-fashioned". Seline was "gentler and shows a little more sympathy for you, which provokes a scowl from the others [...] she is said to have a pretty voice, but completely lacking the range". She survived into adulthood because Helene Raff mentions referring to family records kept by her, and von Bülow records that she and Maria Antonia secured weaving jobs on moving to Ravensburg.
The three sons of Franz Joseph Raff all inherited his aptitude for music. As well as Joachim, the 14-year old Peter was reported by von Bülow "to have a very significant musical talent, singing anything after just one hearing", but it was the middle son Caspar who followed his elder brother by turning musical talent into a profession. In 1844, Franz Joseph and Katharina Raff disapproved so strongly of Joachim abandoning his teaching career for a life in music that his relations with them were strained for the rest of their lives, so when Caspar started out on the same path they did all they could to frustrate his ambitions. At the end of 1850 the 19 year-old resigned from the Catholic teachers' training college in St Gallen and two month's later met and made a good impression on Raff's friend Hans von Bülow, who remarked on Caspar's "fabulous resemblance" to him. Von Bülow felt that Caspar was a very capable musician and recommended that Raff accede to Caspar's request to join him in Weimar, which Raff declined. Four month's later von Bülow came across him again in the company of his family, and witnessed the disapproval, criticism and pressure not to follow his brother's path being brought to bear on him by his mother and elder sister in particular. Perhaps unsurprisingly, he made a less favourable impression this time on von Bülow, but was considering taking up a private tutor's position away from the family. Nothing is known about his life in the next four years, but in 1855 Caspar married Eliza Pfisterer and they immediately emigrated to the USA, settling initially in New York before moving on to Scranton, Pennsylvania. In 1858, by now having Americanised his name to Joseph C. Raff, Caspar moved to Owego, a small town in upstate New York, near Binghamton. The American Raff family eventually grew to one son (another Joseph) and six daughters before Eliza died at the early age of 40. Described in a history of Owego written a few years after his death as "a good violinist, but a better pianist", Caspar established himself as a music teacher, acquired the title of professor and became well known in the community as conductor of a succession of cornet bands. A local newspaper reports his contribution to a wedding: "Prof. Raff's orchestra was present and their sweet music added much to the enjoyment of the occasion. During the evening Prof. Raff played a new piece dedicated to the bride, entitled 'Wedding Pleasures.'" He was the composer of many light piano pieces, some of which in the first half of the 1860s celebrated Union successes in the Civil War. In 1888 he moved from Owego to nearby Binghamton and in 1890 sailed back to Germany, possibly his first visit in 40 years to see his family in Europe. When Caspar Raff died, his obituary in the New York Times of 15 July 1893 read: "Professor Joseph C Raff, an eminent composer and professor of music died at his home in Binghamton, NY Thursday evening. He was sixty two years of age. He was well known in his professional capacity throughout the state."