Grandparent involvement?

Long story but i come from a split family my father remarried less then 12 months after splitting from from my mum his new wife had kids from a previous marriage which he took on as his own as they had no contact with there own father. In the start i was visiting my dad every second weekend but after a couple of years i moved inerstate and only heard from my dad on my birthday and special occasions like Christmas and Easter i was around 12. A couple of years ago i returned to the same state as him and have started my own family only to find that my child gets no attention in fact i could count the amount of times my child has seen his grandfather in his life he is almost 5. Yet his other family and there childen are invited over for dinner once a week and the kids are asked for sleepovers all the time! But when i have tried to organise things in the past they are busy/away/ or have the other grandchildren there i am not asking for baby sitting i would just like my son to be known and to get the same attention and love as the others. My child also dosnt see there fathers grandparents. I feel that my child is missing out on that special time and is beginning to ask why. I have mentioned all this in the past to my father and he knows how i feel and his wife has said he has raised his kids and dosnt want to father mine iam a single mum. Am i being unreasonable and asking to much? Or should i just accept that he has chosen his family and we are not part of it and only a burden that he feels he has to call on birthdays and Christmas And my son will only see him at family events?