Romans 14:16-19 Let not then your good be evil spoken of: For the kingdom of God is not meat and drink; but righteousness, and peace, and joy in the Holy Ghost. For he that in these things serveth Christ is acceptable to God, and approved of men. Let us therefore follow after the things which make for peace, and things wherewith one may edify another.
You are doing right or at least may be but you do not want it to be spoken evil of! The kingdom of God is not about whether I can eat or drink that but righteousness, peace and joy in the Holy Spirit.
The point is to be serving Christ which ever position you take. It is not about you. If you keep the right attitude then you will be doing what God wants and men will approve you.
Instead of being divisive this new attitude will bring peace. We ought to have as a goal to keep the peace among the brethren and in the church. We should be trying to build up our brother not prove our point.
Liberty is a blessing and should not be divisive. We should handle it right. We should be careful of our attitude. We should not be seeking to just have an excuse to do what we want!