Hi Branden,
hi Dave!

Thanks for bringing this issue to a close. I was wondering what I could contribute any further to not let all these words go to waste. :)

I agree a notice in the repo probably is the best resolution.

Cheers
Sven

comment #24:

> > And I will try to
> > think about a proposal on what to do. (I mean, after all, it could
> > boil down to a INSTALL.REPO sentence: "Beware, when on macos and using
> > homebrew, do not forget to set HOMEBREW_ISYSTEM_PATHS during
> > configure.")
>
> It would be more an issue for the "PROBLEMS" file, but yes.  We can
> document our way around it.

This exchange says to me that rather than Rejected or Invalid, it should be reopened and classified as a documentation issue.

I'm resolving this as "Rejected" because apparently this comes down to a difference of opinion between me and the Homebrew developers, or the uchardet, uh, "bottlers" over where that library's header file should be found.

My opinion is that it should be sought here:

/usr/include/uchardet/uchardet.h

That's where Debian keeps it, and therefore probably Ubuntu and between them a whole boatload of derivative distributions.

It is also the reasonable place to keep header files for a library with a version number of "0.0.7" (I explain why in comment #22).  That release was more recent than I expected; the Debian packager prepared it in May 2020.

The uchardet developers may in fact feel differently, and I'm interested to know if they've gone on the record thus.  I'd also add that a good way to document such a decision, along with the rest of the API, is to write a man page for the library.  That uchardet(3) doesn't exist is a bug.

I believe Sven has a workaround documented in this ticket's history; I'd appreciate it if he'd follow up pointing to or summarizing it.  I don't want macOS users of groff to suffer, and I am not pleased that they are adversely affected by the foregoing difference of opinion.  In case it need be said, if I didn't think my view was on solid ground, I'd change it.

Well, within a day of my publicly singing the praises of Savannah's email-based ticket reply feature, it broke.

I've tried twice to update this ticket by that means and nothing happens.

en,

At 2024-09-07T05:06:11-0400, Sven Schober wrote:

> Follow-up Comment #23, bug #66143 (group groff):
> Thanks again for your very long and elaborate answer!

Can't really help myself!  😅

> > I'm assuming that Autoconf doesn't pick it up, hence the failure on
> > macOS to find the uchardet library's header file.
>
> I did not express myself clear enough: I was talking about the
> homebrew build environment. I can enter it via `brew reinstall -i
> groff`. And it was in that env, that I saw these env variables. And
> there-in I saw that /opt/homebrew/include was "put onto the include
> paths list", meaning, a -I/opt/homebrew/include was appended to the
> clang invocation.
>
> I just followed up on the question how this homebrew specific variable
> is creating any effect:
>
>
> …/tmp/groff-20240905-40241-pj7zgj/groff-1.23.0
> $ grep -i 'HOMEBREW_ISYSTEM_PATHS' -R .
> $
>
>
> Nothing. Ok, I'd half expected that. Why would Autoconf consider such
> a homebrew specific variable? But the question remained...

As far as I know, Autoconf doesn't consider it.

It certainly doesn't show up in the Autoconf-generated files on my
system, including "configure", which contains all the feature tests for
the groff build.

> So, homebrew sets HOMEBREW_ISYSTEM_PATHS in its build env and the shim
> puts it onto the include path list. (Btw. googling for
> HOMEBREW_ISYSTEM_PATHS yields exactly two hits for me, none of which
> are documentation, and none of which were helpful to me.)

My hunch is that HOMEBREW_ISYSTEM_PATHS is a blind alley.

> > No, because what if a distribution puts the "uchardet" directory in
> > a weird place that isn't "/usr/include"?
>
> Yes, like, exactly what homebrew does. :) I was speaking only in the
> context of groff/preconv and it's dependency on uchardet (the "whole
> business" wording was unfortunate). (I would not question the
> motivation and legitimacy of pkg-config in general.)
>
> I am slowly getting the impression, that /usr/include could be
> considered as a distribution managed include path.

That meshes with my understanding.

> (And homebrew is trying to emulate that with /opt/homebrew/include.)

Seems likely, given that if they mess around with /usr/include, they
risk their stuff being stomped on by Apple.

> I read up in the FHS [1] on /usr/include:
>
> > This is where all of the system's general-use include files for the
> > C programming language should be placed.
>
> Mhm, by whom? How is general-use defined?

Don't expect too much of FHS.  It was largely put together by
volunteers.  Standards-writing seems to work best as a compensated
activity.  Which is why tech companies avoid underwriting it.  "How is
this going to make us money?"

Neither is it something the Linux Foundation has elected to spend money
on.

> >> I always understood the c/c++ inclusion mechanism to be simply a
> >> matter of trying all include dirs,
> >
> > Where "all" is probably only "/usr/include", yes.  I don't know off
> > the top of my head how strictly ISO C prescribes this.
>
> I was including (haha) in my head also stuff given via -I on the
> command-line here.

That describes a typical compilation just fine.

Standardized processes are typically a very small subset of what one
encounters in practice.  A standard is the intersection of the
relatively small number of things that practitioners can agree upon.

> > One thing I do know about that standard is that libc header files,
> > meaning those specified by the standard and locatable via "#include
> > <whatever.h>", do not actually have to exist as files.
>
> > The compiler is free to interpret such standard header inclusions in
> > a way that they supply some equivalent to the text of such files.
> > This matters for embedded systems, and is in part a consequence of
> > the fact that ISO C does not take on the responsibility of
> > specifying what a file system is.
>
> Ugh. This, eh, ..., I am having a hard time imagining, what this could
> mean in this context. Probably, this concerns stuff like <stdio.h>,
> <string.h> or such, right?

Right.  I said, "libc header files, meaning those specified by the [ISO
C] standard"...

> But for <uchardet.h> it would probably be safe to assume it is a file?

...which "uchardet.h" is not, so yes, I'd expect it to exist as a plain
file.

I said what I did to avoid saying something so general that it would be
false.

> > I reach the opposite conclusion.  It's pretty obvious that the
> > uchardet library API is wholly unspecified, including the location
> > of its header file.  (I do not say undefined--once you do locate
> > "uchardet.h", it has valid C language declarations.)
>
> My problem would not have arisen at least, but that is surely a very
> limited perspective. But also on other distributions, it would have
> worked, as pkg-config would have found uchardet and put the correct
> path on the include path list. That was my thinking.

In my opinion, Homebrew should alter its "uchardet.pc" file to support
groff's expectation about where "uchardet.h" is located, because as your
research showed, other projects share that expectation.

An alternative is for the uchardet project to document their
expectations, thereby establishing a specified API.  If that then
means that groff needs to change (and to be fair, since as I said,
uchardet's de facto API seems to be fossilized, there seems to be little
prospect of it splitting out a new header file soon), then we can do
adapt.

If they need advice in writing a uchardet(3) man page, I'm prepared to
offer it.

> I have to admit, I often have to read your answers multiple times to
> extract all your meaning from it. :) I will do that. And I will try to
> think about a proposal on what to do. (I mean, after all, it could
> boil down to a INSTALL.REPO sentence: "Beware, when on macos and using
> homebrew, do not > forget to set HOMEBREW_ISYSTEM_PATHS during
> configure.")

It would be more an issue for the "PROBLEMS" file, but yes.  We can
document our way around it.

Regards,
Branden

> {savane: user = 108747; tracker = bugs; item = 66143}

At 2024-09-06T15:03:52-0400, Sven Schober wrote:

> Follow-up Comment #21, bug #66143 (group groff):
>
> So, me again: I found the problem! I called make with CPPFLAGS=-v,
> which displays the exact command line that is executed, in both my
> environments (homebrew and outside).
>
> Inside the homebrew environment, there is a single additional include
> path added: /opt/homebrew/include
>
> And lo and behold:
>
>
> ls -ahl /opt/homebrew/include/ | grep uchar
> lrwxr-xr-x    1 svenschober  admin    41B Feb 13  2023 uchardet ->
> ../Cellar/uchardet/0.0.8/include/uchardet

Aha!

> That is a symlink to the other place, where uchardet is installed,
> and more importantly, that is why an include of the form
> `<uchardet/uchardet.h>` is working:
> `/opt/homebrew/include/uchardet/uchardet.h` is existant.
>
> But not when that include path is missing.

Right.

> I can add that path via:
>
> ../configure --without-x --with-uchardet CPPFLAGS="-I/opt/homebrew/include"
>
>
> But, I ask myself, why this is not necessary in the homebrew env.
>
> I found the following:
>
>
> env | grep include
> HOMEBREW_INCLUDE_PATHS=/opt/homebrew/opt/readline/include:/opt/homebrew/opt/sqlite/include
> HOMEBREW_ISYSTEM_PATHS=/opt/homebrew/include:/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
>
> It seems, they set a variable called HOMEBREW_ISYSTEM_PATHS. But how configure
> gets to pick that up is still a riddle to me.

I'm assuming that Autoconf doesn't pick it up, hence the failure on
macOS to find the uchardet library's header file.

> But doesn't this mean, my point still stands: the whole pkg-config
> business is superfluous,

No, because what if a distribution puts the "uchardet" directory in a
weird place that isn't "/usr/include"?

> as long as the cpp source references
> `<uchardet/uchardet.h>`?

That just moves the problem.  "uchardet.h" or "uchardet/uchardet.h", if
the system doesn't supply it in /usr/include, odds are that pkg-config
will be necessary to locate it.  And nobody, as far as I know is so
prescriptivist about the Filesystem Hierarchy Standard (FHS) that they
require that .h files have names resolvable only from that directory.

> I always understood the c/c++ inclusion mechanism to be simply a
> matter of trying all include dirs,

Where "all" is probably only "/usr/include", yes.  I don't know off the
top of my head how strictly ISO C prescribes this.

One thing I do know about that standard is that libc header files,
meaning those specified by the standard and locatable via "#include
<whatever.h>", do not actually have to exist as files.

The compiler is free to interpret such standard header inclusions in a
way that they supply some equivalent to the text of such files.  This
matters for embedded systems, and is in part a consequence of the fact
that ISO C does not take on the responsibility of specifying what a file
system is.

> one by one and append the inclusion path.

For just about anything that uses any library that isn't libc, the
specification of `-I` is somewhere from strongly encouraged to
mandatory.  /usr/include itself has been getting more crowded with plain
files for decades; as a result the name space collision problem
re-asserts itself.

> If there is no match, we get the error I am getting.  But I could be
> wrong.

I think you're mostly on track.

> I did some further digging and found another project using uchardet
> via uchardets website:
>
> https://github.com/mpv-player/mpv/blob/0eb5e914d9699b6f7fb91ee383dedadc491b0d7d/misc/charset_conv.c#L67
>
> They also include it without the directory. In this old branch of
> another project the same:
>
> https://gitlab.gnome.org/World/gedit/libgedit-tepl/-/blob/tepl-3-0/tepl/tepl-file-loader.c
>
> But, I also found counter examples (using the qualified form):
>
> https://github.com/wang-bin/QtAV/blob/8bb780215bcd4a16d098a2a913d01f83b16193d7/config.tests/uchardet/main.cpp#L21
>
> and
>
> https://github.com/eranif/codelite/blob/345332fca8ae6d29ca6615dccec81d92525bd6f1/LiteEditor/cl_editor.cpp#L102

Right.  The uchardet authors/maintainers didn't document their API, so
people took guesses.  I suspect that users of that API selected whatever
happened to work on their system, if the first try didn't work, the
second usually did, so "must have been right".

> In summary, I don't know. I now know how to fix my build problem, but
> I have a hunch, that removing the directory prefix would be a
> portability win.

I reach the opposite conclusion.  It's pretty obvious that the
uchardet library API is wholly unspecified, including the location of
its header file.  (I do not say undefined--once you do locate
"uchardet.h", it has valid C language declarations.)

This unspecified status is common in the early stages of library
development.  The developers don't want to be wedded to mistaken
decisions or held responsible for support of aspects of the interface
that they had reason to reconsider.  They want to be free to delete
functions, change the arguments they take, rename data types, and even
merge or split header files.

Importantly, if the uchardet developers had decided in early days to
split "uchardet.h" into multiple files--maybe they'd want a "uchar.h"
file too--it would be a pain in the butt to go and try to hunt up name
collisions to see if they could claim a particular name.

I reckon that at least some package maintainers, especially those with
hard experience in GNU/Linux distributions, appreciated that the API was
in flux, and so they gave uchardet's header file a sandbox directory.

If someone were to ask:

"Why would you have a subdirectory of /usr/include for just one header
file with the same name (barring the .h suffix) as the directory
itself?"

That'd be why.  People downstream of the uchardet developers had no
idea what the future would hold.

It appears that uchardet began life as a paid development project of
employees of the Mozilla Corporation.  It's possible that these
developers were suddenly "made redundant" and thus "uchardet.h"
"stabilized"--by fossilizing as it was when the pink slips came down,
like the half-finished breakfasts and open newspapers on the kitchen
tables of Pripyat, Ukraine, in April 1986.

That's tech sector capitalism for you--destruction is creative and while
you can't turn a ship on a dime, you can fire people so fast that the
optical Doppler shift is visible to the naked eye.

> Otherwise, why use pkg-config then after all?

To an extent, pkg-config insulates software dependent on a library
from disruptive changes to that library's interface at the coarsest
level (where the header files and library objects are, and what they are
named).

There was, and is, an alternative--writing Autoconf tests to figure
these things out.  But few people find that more reliable, or enjoyable.

Regards,
Branden

Hi Branden!

Sorry to keep you so busy with my strange problem and or setup.

But at least I am learning new stuff! I did not know about:
 

make V=1 preconv

That is great, and there are many situations, where I wish I had known this.

So, I went ahead and investigated the homebrew build process some more. Turns it, there is a way to download the sources and get a shell just before the recipe, or formula how they call it, is being executed. So, I could follow the recipe manually and inspect what is happening.

First observation, the sources build untouched (surprise). This is what I see:

…/tmp/groff-20240905-40241-pj7zgj/groff-1.23.0
$  touch src/preproc/preconv/preconv.cpp

…/tmp/groff-20240905-40241-pj7zgj/groff-1.23.0
$ make V=1 preconv
clang++ -DHAVE_CONFIG_H -I. -I./src/include  -I./src/include -I./lib -I./src/include -I./lib -I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet   -g -O2 -MT src/preproc/preconv/preconv-preconv.o -MD -MP -MF src/preproc/preconv/.deps/preconv-preconv.Tpo -c -o src/preproc/preconv/preconv-preconv.o `test -f 'src/preproc/preconv/preconv.cpp' || echo './'`src/preproc/preconv/preconv.cpp
mv -f src/preproc/preconv/.deps/preconv-preconv.Tpo src/preproc/preconv/.deps/preconv-preconv.Po
clang++  -g -O2   -o preconv src/preproc/preconv/preconv-preconv.o libgroff.a -lm -liconv -L/opt/homebrew/Cellar/uchardet/0.0.8/lib -luchardet lib/libgnu.a

So, clang++ is the compiler and pkg-config output is added to the include paths.

Then, I wanted to see which headers are really considered and learned about the -H flag:

CPPLAGS=-H make V=1 CPPFLAGS=-H 2>&1 | grep uchardet
clang++ -DHAVE_CONFIG_H -I. -I./src/include  -I./src/include -I./lib -I./src/include -I./lib -I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet -H  -g -O2 -MT src/preproc/preconv/preconv-preconv.o -MD -MP -MF src/preproc/preconv/.deps/preconv-preconv.Tpo -c -o src/preproc/preconv/preconv-preconv.o `test -f 'src/preproc/preconv/preconv.cpp' || echo './'`src/preproc/preconv/preconv.cpp
. /opt/homebrew/include/uchardet/uchardet.h
clang++  -g -O2   -o preconv src/preproc/preconv/preconv-preconv.o libgroff.a -lm -liconv -L/opt/homebrew/Cellar/uchardet/0.0.8/lib -luchardet lib/libgnu.a

The line with the dot shows, that indeed the uchardet.h put on the include path is taken.

Then, I inspected with my newly acquired V=1 skill, what is happening on my machine (outside of the homebrew build and again with unaltered sources):

make V=1 preconv
g++ -std=gnu++11 -DHAVE_CONFIG_H -I. -I.. -I./src/include  -I../src/include -I../lib -I./src/include -I./lib -I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet   -g -O2 -MT src/preproc/preconv/preconv-preconv.o -MD -MP -MF src/preproc/preconv/.deps/preconv-preconv.Tpo -c -o src/preproc/preconv/preconv-preconv.o `test -f 'src/preproc/preconv/preconv.cpp' || echo '../'`src/preproc/preconv/preconv.cpp
../src/preproc/preconv/preconv.cpp:30:10: fatal error: 'uchardet/uchardet.h' file not found
#include <uchardet/uchardet.h>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [src/preproc/preconv/preconv-preconv.o] Error 1

AHA! The compiler is g++ (I remember that I installed it, when I was experimenting with c++ modules some time ago)!

So, what happens when I switch this to clang++?:

make V=1 CXX=clang++ preconv
clang++ -DHAVE_CONFIG_H -I. -I.. -I./src/include  -I../src/include -I../lib -I./src/include -I./lib -I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet   -g -O2 -MT src/preproc/preconv/preconv-preconv.o -MD -MP -MF src/preproc/preconv/.deps/preconv-preconv.Tpo -c -o src/preproc/preconv/preconv-preconv.o `test -f 'src/preproc/preconv/preconv.cpp' || echo '../'`src/preproc/preconv/preconv.cpp
../src/preproc/preconv/preconv.cpp:30:10: fatal error: 'uchardet/uchardet.h' file not found
#include <uchardet/uchardet.h>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [src/preproc/preconv/preconv-preconv.o] Error 1

So, this is now the time when I am slowly starting to doubt my sanity. I don't see any differences in the invocations. The clang++ versions called outside homebrew and inside are the same.

Sorry, disordered recipe in comment #17.  (I had some false starts in my shell history and edited it clumsily.)

$ rm -rf build
$ mkdir build
$ ./bootstrap
$ cd build
$ UCHARDET_CFLAGS=-I/hi/sven/here/is/an/example ../configure
$ make -j
$ rm preconv ./src/preproc/preconv/*.o
$ make V=1 preconv

Hi Branden!

Thank you for your long answer and background information on C and C++!

> To be honest, I didn't try,

I am a bit sad - but I fully understand your hesitancy.

I will try and channel my emotional energy into trying to find out, why and how the homebrew build of groff works. I have a suspicion it is related to setting the prefix explicitly [1]. I am hard pressed of guessing, what that variable is set tot during build, so I will have to try it out.

> because I've concluded that Homebrew is misconfiguring uchardet's pkg-config output.

Honestly, I do not fully understand, what you mean by this. But maybe my investigation of homebrews own build process will further my understanding.

> Feel free to direct Homebrew developers to this ticket.  Maybe I'm wrong.

My goal is simply to be able to build groff from source on macos with homebrew without needing to patch the groff sources. If it is homebrew, which is at fault, I will certainly open a ticket there. :)

Thank you again for the time you take to address my problem!

[1]: https://github.com/Homebrew/homebrew-core/blob/ece78c067d1462730eb0602077316f5ce76d15ab/Formula/g/groff.rb#L39

Hi Branden!

To answer your questions first:

grep UCHARDET_ config.status
S["UCHARDET_LIBS"]="-L/opt/homebrew/Cellar/uchardet/0.0.8/lib -luchardet"
S["UCHARDET_CFLAGS"]="-I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet"

and

pkg-config --cflags   "uchardet"
-I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet
pkg-config --libs   "uchardet"
-L/opt/homebrew/Cellar/uchardet/0.0.8/lib -luchardet

So, I think this is equivalent to your path, just with a different prefix.

But, in the mean time I no longer think pkg-config, or autotools is at fault here.

My somewhat bold claim would be, that the #include statement with a subdirectory is simply wrong and works only accidentally on systems where /usr/include is on the default search path.

You could check that on your system by simply removing the subdirectory from the include in src/preproc/preconv/preconv.cpp (I execute the following commands in my build subdir):

sed -i -e 's;uchardet/;;' ../src/preproc/preconv/preconv.cpp
make
make
/Library/Developer/CommandLineTools/usr/bin/make  all-recursive
  CXX      src/preproc/preconv/preconv-preconv.o
  CXXLD    preconv
  GROFF    doc/meintro_fr.ps

or on debian

sed -i -e 's;uchardet/;;' ../src/preproc/preconv/preconv.cpp
svenschober@debian:~/src/groff/build$ make
make  all-recursive
make[1]: Entering directory '/home/svenschober/src/groff/build'
make[2]: Entering directory '/home/svenschober/src/groff/build'
  CXX      src/preproc/preconv/preconv-preconv.o
  CXXLD    preconv
  GROFF    doc/meintro_fr.ps
make[2]: Leaving directory '/home/svenschober/src/groff/build'
make[1]: Leaving directory '/home/svenschober/src/groff/build'

Works fine on both systems, as only now the pkg-config generated includes are heeded. What do you think? Can you reproduce this on your systems?

comment #10:

> > The dueling-pdftotexts problem should likely be a separate bug report.
>
> Sorry for the mess

Not at all; this is how bug reports are supposed to work.  You report your findings, but without enough info to determine whether they're related.  Subsequent research finds they're not, so new reports are opened to cover them.  I've made plenty of such "messes" in my time here.

I went ahead and opened a new ticket for the pdftotext issue: bug #66155.

> I thought about a configure test, issuing a warning if the
> wrong/missbehaving xpdf tools are found?

I don't think a tool working differently from what groff happened to expect is wrong or misbehaving.  It's just an unfortunate (but also unsurprising, given the nature of the command) name collision between different contributors' implementations of the same basic functionality.  This to me isn't something that needs to be warned about.

> The dueling-pdftotexts problem should likely be a separate bug report.

Sorry for the mess - but, in my defense, I was a bit hesitant in the beginning.. :)

> Solving (2) is a matter of, I think, having the test scripts check the program behavior and SKIPping the test if the program doesn't behave the way we expect.

I thought about a configure test, issuing a warning if the wrong/missbehaving xpdf tools are found?

> We have (1) a possible pkg-config problem on macOS

ATM, I only have access to macos, otherwise I would try to compare the outcomes of the different pkg-configs on macos and archlinux, e.g.. Could it be also the way uchardet is installed  via homebrew?

brew list uchardet | grep '\.h'
/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet/uchardet.h

comment #7:

> The underlying problem seems to be...

To clarify: the underlying problem to this test issue, not to the build issue that is this bug report's primary focus.  The dueling-pdftotexts problem should likely be a separate bug report.

For my uchardet header include problem: My package config file reads like this:

cat /opt/homebrew/lib/pkgconfig/uchardet.pc
libdir=/opt/homebrew/Cellar/uchardet/0.0.8/lib
includedir=/opt/homebrew/Cellar/uchardet/0.0.8/include

Name: uchardet
Description: An encoding detector library ported from Mozilla
Version: 0.0.8
Requires:
Libs: -L${libdir} -luchardet
Libs.private: -lstdc++
Cflags: -I${includedir}/uchardet

And it seems, this is unaltered from the sources [1]. My pkg-config directly picks this up and emits the following cflags:

Makefile
3876:UCHARDET_CFLAGS = -I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet

I am no expert on clang invocations and header search path semantics, but with that include string it seems natural for me that this is failing.

What am I missing?

[1]: https://gitlab.freedesktop.org/uchardet/uchardet/-/blob/master/uchardet.pc.in?ref_type=heads

> Just a quick note, an "XFAIL" is an expected failure...

Aha! Oh man, this is a bit embarassing.. I thought it meant execution failure or something... and I did not take the time to investigate the calling code. Sorry for wasting your time on that.

There is one other real FAIL left, which seems to be connected to not behaving as expected, right?:

...
checking HTML output of web URI
pdftotext version 4.05 [www.xpdfreader.com]
Copyright 1996-2024 Glyph & Cog, LLC
Usage: pdftotext [options] <PDF-file> [<text-file>]
  -f <int>               : first page to convert
  -l <int>               : last page to convert
  -layout                : maintain original physical layout
  -simple                : simple one-column page layout
...

I think my version of pdftotext does not support reading the PDF from stdin?

> I've not aware of uchardet library detection being broken anywhere else, and there are no recent reports of such breakage that I can remember.

I do not think it's 'broken' -- it's more like too exact, or specific?

$ rg uchardet
....
config.status
...
685:S["UCHARDET_LIBS"]="-L/opt/homebrew/Cellar/uchardet/0.0.8/lib -luchardet"
686:S["UCHARDET_CFLAGS"]="-I/opt/homebrew/Cellar/uchardet/0.0.8/include/uchardet"
...

Here, this line 686: The added include path already contains the subfolder, so there is no need to specify it in the header file.

Anyway, I think, I was being too ominous with my hint, that I am seeing test failures.

But to first do what you asked me for:

./preconv -v
GNU preconv (groff) version 1.23.0.1842-c9b3c with iconv support and with uchardet support

I think preconv is working fine.

I've attached my test-suite.log file, so you can see what is going on for me.

Those mom tests failing are not related to preconv not functioning correctly, but actually one is caused by the version of pdfimages on my machine takes a different amount of arguments, than what seems to be expected in the test case. This is what I get in the test-suite.log:

...
FAIL: contrib/mom/examples/tests-mom.sh
=======================================

Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/letter.pdf
Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/mom-pdf.pdf
Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/mon_premier_doc.pdf
Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/sample_docs.pdf
Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/slide-demo.pdf
Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/typesetting.pdf
Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/copyright-chapter.pdf
Checking number of pages of /Users/svenschober/src/groff/build/contrib/mom/examples/copyright-default.pdf
Checking if /Users/svenschober/src/groff/build/contrib/mom/examples/typesetting.pdf has images
pdfimages version 4.05 [www.xpdfreader.com]
Copyright 1996-2024 Glyph & Cog, LLC
Usage: pdfimages [options] <PDF-file> <image-root>
  -f <int>         : first page to convert
  -l <int>         : last page to convert
  -j               : write JPEG images as JPEG files
  -raw             : write raw data in PDF-native formats
  -list            : write information to stdout for each image
  -opw <string>    : owner password (for encrypted files)
  -upw <string>    : user password (for encrypted files)
  -verbose         : print per-page status information
  -q               : don't print any messages or errors
  -cfg <string>    : configuration file to use in place of .xpdfrc
  -v               : print copyright and version info
  -h               : print usage information
  -help            : print usage information
  --help           : print usage information
  -?               : print usage information
 no images found
...

My version of pdfimages is:

pdfimages -v
pdfimages version 4.05 [www.xpdfreader.com]
Copyright 1996-2024 Glyph & Cog, LLC

And I've installed it via homebrew and the xpdf package:

brew info xpdf
==> xpdf: stable 4.05 (bottled)
PDF viewer
https://www.xpdfreader.com/
Conflicts with:
  pdf2image (because poppler, pdftohtml, pdf2image, and xpdf install conflicting executables)
  pdftohtml (because poppler, pdftohtml, pdf2image, and xpdf install conflicting executables)
  poppler (because poppler, pdftohtml, pdf2image, and xpdf install conflicting executables)
Installed
/opt/homebrew/Cellar/xpdf/4.05 (26 files, 14.9MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-02-16 at 20:11:31
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/x/xpdf.rb
License: GPL-2.0-only OR GPL-3.0-only
==> Dependencies
Build: cmake ✔
Required: fontconfig ✔, freetype ✔, libpng ✔, qt@5 ✔
==> Analytics
install: 316 (30 days), 977 (90 days), 5,427 (365 days)
install-on-request: 316 (30 days), 977 (90 days), 5,427 (365 days)
build-error: 0 (30 days)

I can fix this, adding an image root argument of 1 to the invocation and then I get:

...
Checking if /Users/svenschober/src/groff/build/contrib/mom/examples/typesetting.pdf has images
 no images found
Checking if /Users/svenschober/src/groff/build/contrib/mom/examples/slide-demo.pdf has images
 no images found
...

Hmm... I check what pdfimage is giving me on those files:

pdfimages -list contrib/mom/examples/typesetting.pdf 1
1-0000.ppm: page=3 width=81 height=96 hdpi=72.00 vdpi=72.00 colorspace=DeviceRGB bpc=8
1-0001.pgm: page=3 width=81 height=96 hdpi=72.00 vdpi=72.00 colorspace=DeviceGray bpc=8

This seems now to be caused by pdfimages spitting out two lines and the test expects there to be more (?). I check the file and I see only one image on page three to be sure.

Now, again, I don't know the expectations of the test and the results, but I can change that to -lt and the test passes. :)

So, my changes to the test would be:

git diff
diff --git a/contrib/mom/examples/test-mom.sh.in b/contrib/mom/examples/test-mom.sh.in
index 21c8835b8..e2b1364da 100644
--- a/contrib/mom/examples/test-mom.sh.in
+++ b/contrib/mom/examples/test-mom.sh.in
@@ -63,8 +63,8 @@ check_number_pages()
 check_has_images()
 {
     echo "Checking if $1 has images"
-    n_lines=`pdfimages -list $1 | wc -l `
-    if test $n_lines -le 2; then
+    n_lines=`pdfimages -list $1 1 | wc -l `
+    if test $n_lines -lt 2; then
         echo " no images found"
         ret=255
     fi

But continuing from there, there are further failures.

There is one about unicode escaping:

...
XFAIL: src/roff/groff/tests/string_case_xform_unicode_escape.sh
===============================================================

XFAIL src/roff/groff/tests/string_case_xform_unicode_escape.sh (exit status: 1)
...

Now, this is not terribly helpful, so I added a set -x to that script and got:

XFAIL: src/roff/groff/tests/string_case_xform_unicode_escape.sh
===============================================================

+ groff=/Users/svenschober/src/groff/build/test-groff
+ expected='attaché ATTACHÉ'
++ /Users/svenschober/src/groff/build/test-groff -Tutf8
+ actual='troff:<standard input>:5: warning: special character '\''U0065_0301'\'' not defined
attaché ATTACH'
+ echo 'troff:<standard input>:5: warning: special character '\''U0065_0301'\'' not defined
attaché ATTACH'
+ grep -Fqx 'attaché ATTACHÉ'
XFAIL src/roff/groff/tests/string_case_xform_unicode_escape.sh (exit status: 1)

So, if I interpret this correctly, groff is issuing a warning, which is then part of the string to be compared and thus the comparison fails...

I can continue like this, and I kindly ask you, if you think, this is time well spent?

Would you be interessted in a short write-up of my experiences of building groff on macos?

(file #56398)

Hi!

I am currently trying to build the HEAD/master (c9b3c99) on my MacOS 14.6.1 machine using a lot of homebrew installed deps (so I am always willing to admit, this might be due to a botched environment if mine caused by historical accumulation of cruft).

I followed INSTALL.REPO and boostrapped the repo successfully.

When issuing the make command, I get the following error:

$ make
/Library/Developer/CommandLineTools/usr/bin/make  all-recursive
  CXX      src/preproc/preconv/preconv-preconv.o
../src/preproc/preconv/preconv.cpp:30:10: fatal error: 'uchardet/uchardet.h' file not found
#include <uchardet/uchardet.h>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [src/preproc/preconv/preconv-preconv.o] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Which is easily solvable for me, by issuing the following patch:

diff --git a/src/preproc/preconv/preconv.cpp b/src/preproc/preconv/preconv.cpp
index cc661c525..8a13988a4 100644
--- a/src/preproc/preconv/preconv.cpp
+++ b/src/preproc/preconv/preconv.cpp
@@ -27,7 +27,7 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>. */
 #include <errno.h>
 #include <sys/stat.h>
 #ifdef HAVE_UCHARDET
-#include <uchardet/uchardet.h>
+#include <uchardet.h>
 #endif

 #include "errarg.h"

But I am not sure, why this is necessary.

I had the suspicion, that this might be due to pkg-config? My version is installed via homebrew:

brew info pkg-config
==> pkg-config: stable 0.29.2 (bottled)
Manage compile and link flags for libraries
https://freedesktop.org/wiki/Software/pkg-config/
Conflicts with:
  pkgconf (because both install `pkg.m4` file)
Installed
/opt/homebrew/Cellar/pkg-config/0.29.2_3 (12 files, 679.3KB)
  Poured from bottle using the formulae.brew.sh API on 2024-07-19 at 19:20:48
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/pkg-config.rb
License: GPL-2.0-or-later
==> Analytics
install: 63,483 (30 days), 188,030 (90 days), 642,084 (365 days)
install-on-request: 16,770 (30 days), 50,447 (90 days), 183,614 (365 days)
build-error: 342 (30 days)

My installed version of uchardet is 0.0.8 also via homebrew:

brew info uchardet
==> uchardet: stable 0.0.8 (bottled), HEAD
Encoding detector library
https://www.freedesktop.org/wiki/Software/uchardet/
Installed
/opt/homebrew/Cellar/uchardet/0.0.8 (17 files, 652.3KB) *
  Poured from bottle on 2023-02-13 at 14:47:25
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/u/uchardet.rb
License: MPL-1.1 OR GPL-2.0-or-later OR LGPL-2.1-or-later
==> Dependencies
Build: cmake ✔
==> Options
--HEAD
        Install HEAD version
==> Analytics
install: 1,172 (30 days), 3,842 (90 days), 15,415 (365 days)
install-on-request: 120 (30 days), 424 (90 days), 1,698 (365 days)
build-error: 0 (30 days)

Please let me know, if you need any further information about my system (I've attached the config.log, which might help).

As a sidenote: With my "patch" above everything builds just fine, but I am seeing failed tests, when issuing make check, e.g., in mom tests. I could solve some of them using relatively harmless changes to contrib/mom/examples/test-mom.sh.in.

Should I create a separate issue for the test failures?