Android RecyclerView OnScrollListener onScrolled function and ViewBinding visibility issue

Question

I'm experiencing weird behavior with the following code when trying to show an ImageButton on screen that scrolls a RecyclerView back up top.

        myViewManager = LinearLayoutManager(context)
        val minimumScrollPosition = if(resources.getBoolean(R.bool.is_tablet)) 12 else 7

        binding.myRecyclerview.apply {
            layoutManager = myViewManager
            setHasFixedSize(true)
            addOnScrollListener(object : RecyclerView.OnScrollListener() {
                override fun onScrolled(recyclerView: RecyclerView, dx: Int, dy: Int) {
                    Log.d(TAG, "Position scrolled: ${(recyclerView.layoutManager as LinearLayoutManager).findFirstVisibleItemPosition()}")
                    if((recyclerView.layoutManager as LinearLayoutManager).findFirstVisibleItemPosition() >= minimumScrollPosition) {
                        binding.scrollToTopButton.visibility = View.VISIBLE
                    } else {
                        binding.scrollToTopButton.visibility = View.GONE
                    }
                }
            })
        }

The adapter for this recyclerview is set at a later portion of the code; this all resides inside the Fragment's onViewCreated function. While this does work on a first visit to the fragment, if I were to navigate backwards from a page ahead(a details page when tapping on a row inside the recyclerview), the code does satisfy that condition, but is unable to make the imagebutton visible with the viewbinding; this also applies if I were to use view.findViewById approach as well.

There is also weird behavior where following this, the button does appear via this onScrollListener onScrolled function callback if the recyclerview sees findFirstVisibleItemPosition as the minimumScrollPosition at least once; if I were to navigate backwards and have the recyclerview start off at a position where it was tapped before navigating forward, the button won't appear despite satisfying the condition inside onScrolled function callback.

I would like for someone to explain this issue to me, and any steps I can take to remedy it.