Use GitHub Actions for builds

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Use GitHub Actions for builds

Michael Mattioli
I pushed a commit to my fork which replaces Travis and AppVeyor with GitHub
Actions for builds. You can see the commit here:
https://github.com/mmattioli/krb5/commit/3ecd36d4b701f2e6f01f6148c51d246ad001880f

Why? Was curious to see if it would work and how well it would work.
Wondering if there's an appetite to help me work through some bugs and
replace them or not. Just curious. Some notes:

- I had to remove the installation of slapd on the linux builds because the
package it was trying to install was asking for user input which isn't
possible in an automated environment. Despite removing it from
installation, there are no errors in the three linux build variants.
- The Windows build is failing at what seems to be a syntax error
somewhere? The output from the log where it seems to be failing is from
this command: perl -w .\../util/def-check.pl private-and-public-decls
.\../lib/krb5_32.def

2019-12-02T03:50:45.1330582Z Microsoft (R) Program Maintenance Utility
Version 14.23.28106.4
2019-12-02T03:50:45.1331518Z Copyright (C) Microsoft Corporation.  All
rights reserved.
2019-12-02T03:50:45.1331878Z
2019-12-02T03:50:45.1383429Z copy/b nul:+ win-mac.h autoconf.h
2019-12-02T03:50:45.1564127Z nul
2019-12-02T03:50:45.1570480Z win-mac.h
2019-12-02T03:50:45.1890748Z         1 file(s) copied.
2019-12-02T03:50:45.1907748Z copy/b nul:+ osconf.hin osconf.h
2019-12-02T03:50:45.2068478Z nul
2019-12-02T03:50:45.2072899Z osconf.hin
2019-12-02T03:50:45.2079706Z         1 file(s) copied.
2019-12-02T03:50:45.2095011Z copy /y
krb5\krb5.h+k5-int.h+krb5\clpreauth_plugin.h private-and-public-decls
2019-12-02T03:50:45.2251785Z krb5\krb5.h
2019-12-02T03:50:45.2265156Z k5-int.h
2019-12-02T03:50:45.2268641Z krb5\clpreauth_plugin.h
2019-12-02T03:50:45.2273313Z         1 file(s) copied.
2019-12-02T03:50:45.2287382Z perl -w .\../util/def-check.pl
private-and-public-decls .\../lib/krb5_32.def
2019-12-02T03:50:46.4356046Z Invalid function name: '
2019-12-02T03:50:46.4356385Z
2019-12-02T03:50:46.4356597Z
2019-12-02T03:50:46.4356745Z  as in DCE, or for users */
2019-12-02T03:50:46.4356929Z  */
2019-12-02T03:50:46.4357105Z  NT_WELLKNOWN principals */
2019-12-02T03:50:46.4358065Z
2019-12-02T03:50:46.4359329Z
2019-12-02T03:50:46.4359706Z  typedef const krb5_principal_data
*krb5_const_principal
2019-12-02T03:50:46.4359905Z
2019-12-02T03:50:46.4360478Z ' at .\../util/def-check.pl line 175, <GEN0>
line 258.
2019-12-02T03:50:46.4598010Z NMAKE : fatal error U1077: '"C:\Program
Files\Git\usr\bin\perl.EXE"' : return code '0xff'
2019-12-02T03:50:46.4598317Z Stop.
2019-12-02T03:50:46.4632550Z NMAKE : fatal error U1077: 'for' : return code
'0x1'
2019-12-02T03:50:46.4632877Z Stop.


- You can checkout the 4x build runs here:
https://github.com/mmattioli/krb5/commit/1c6122ea55f7ec5f681a8bcc7e825bd4e20b5207/checks?check_suite_id=337012953

Mike
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Use GitHub Actions for builds

Greg Hudson
On 12/1/19 11:16 PM, Michael Mattioli wrote:
> I pushed a commit to my fork which replaces Travis and AppVeyor with GitHub
> Actions for builds.

I was just starting to look into Github Actions for krb5 over the long
weekend weekend, partly to see if I could be more successful at setting
up a MacOS job than I was with Travis.  Having a mostly-complete
baseline will save a lot of work.

If tests runs significantly faster than they do under Travis/Appveyor,
that may be a reason to switch over.  If they don't, and I can't get a
MacOS job working, I don't know if there will be a good reason to switch.

> - I had to remove the installation of slapd on the linux builds because the
> package it was trying to install was asking for user input which isn't
> possible in an automated environment. Despite removing it from
> installation, there are no errors in the three linux build variants.

t_kdb.py detects that slapd is not present and skips those tests.  The
skiptest contents in the builds generated by your branch says:

Skipped URI discovery tests: resolv_wrapper 1.1.5 not found
5808
Skipped LDAP KDB tests: slapd not found

while the Travis integration currently doesn't skip anything.  I will
look into why slapd can't be installed and why resolv_wrapper isn't found.

(Ever since we added "make clean" checking to the Travis job, it's a lot
harder to see what tests are skipped.  I might make the CI script
re-display the skiptests file at the end of the script to fix that.)

> - The Windows build is failing at what seems to be a syntax error
> somewhere? The output from the log where it seems to be failing is from
> this command: perl -w .\../util/def-check.pl private-and-public-decls
> .\../lib/krb5_32.def

This is probably a newline issue, since it's complaining about a
function name spanning multiple lines of text.  github's Windows runner
may have a preinstalled git with different default behavior.  I will
look into that as well.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Use GitHub Actions for builds

Michael Mattioli
> I was just starting to look into Github Actions for krb5 over the long
> weekend weekend, partly to see if I could be more successful at setting
> up a MacOS job than I was with Travis.

What compiler, flags, etc. did you want to test on macOS? They only provide the latest version of macOS (10.15, Catalina). I can give it a spin and see if it works.

Mike
Sent from my iPhone

> On Dec 2, 2019, at 1:11 AM, Greg Hudson <[hidden email]> wrote:
>
> ´╗┐On 12/1/19 11:16 PM, Michael Mattioli wrote:
>> I pushed a commit to my fork which replaces Travis and AppVeyor with GitHub
>> Actions for builds.
>
> I was just starting to look into Github Actions for krb5 over the long
> weekend weekend, partly to see if I could be more successful at setting
> up a MacOS job than I was with Travis.  Having a mostly-complete
> baseline will save a lot of work.
>
> If tests runs significantly faster than they do under Travis/Appveyor,
> that may be a reason to switch over.  If they don't, and I can't get a
> MacOS job working, I don't know if there will be a good reason to switch.
>
>> - I had to remove the installation of slapd on the linux builds because the
>> package it was trying to install was asking for user input which isn't
>> possible in an automated environment. Despite removing it from
>> installation, there are no errors in the three linux build variants.
>
> t_kdb.py detects that slapd is not present and skips those tests.  The
> skiptest contents in the builds generated by your branch says:
>
> Skipped URI discovery tests: resolv_wrapper 1.1.5 not found
> 5808
> Skipped LDAP KDB tests: slapd not found
>
> while the Travis integration currently doesn't skip anything.  I will
> look into why slapd can't be installed and why resolv_wrapper isn't found.
>
> (Ever since we added "make clean" checking to the Travis job, it's a lot
> harder to see what tests are skipped.  I might make the CI script
> re-display the skiptests file at the end of the script to fix that.)
>
>> - The Windows build is failing at what seems to be a syntax error
>> somewhere? The output from the log where it seems to be failing is from
>> this command: perl -w .\../util/def-check.pl private-and-public-decls
>> .\../lib/krb5_32.def
>
> This is probably a newline issue, since it's complaining about a
> function name spanning multiple lines of text.  github's Windows runner
> may have a preinstalled git with different default behavior.  I will
> look into that as well.

_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Use GitHub Actions for builds

Greg Hudson
On 12/2/19 6:12 AM, Michael Mattioli wrote:
> What compiler, flags, etc. did you want to test on macOS? They only provide the latest version of macOS (10.15, Catalina). I can give it a spin and see if it works.

clang, and the same flags as the first Ubuntu clang job.  I tried to set
this up with Travis a couple of years ago and ran into two issues:

1. It was too slow--generally about 15 minutes to run a build and check
instead of 7-8.  Github Actions uses the same macOS hosting provider as
Travis (Macstadium), so might have the same problem.  But it's worth a try.

2. The macOS images have System Integrity Protection enabled, which
interferes with the test suite.  The most significant problem is that
the shell clears DYLD_LIBRARY_PATH from the environment.  There also
appear to be some signal restrictions which break the iprop tests.  My
attempts to work around these problems got as far as
https://github.com/greghudson/krb5/commit/f99f4e9c79372516a8905e840e9ddc696c97c81c
but the make-host-keytab.plin change is broken; it needs to suppress
output and ignore errors when running one of the kadmin ank commands,
without using the shell, and I couldn't find an easy way to do that in
Perl.  That's a surmountable problem, but I got frustrated with the slow
builds and stopped at that point.  It's possible that Github Actions
turns off SIP on its macOS runners; otherwise we'll need the same
workarounds.

[I wrote:]
>> while the Travis integration currently doesn't skip anything.  I will
>> look into why slapd can't be installed and why resolv_wrapper isn't found.

slapd can be installed if we do "sudo DEBIAN_FRONTEND=noninteractive
apt-get ...".  Travis arranges for the noninteractive front-end by
default.  I submitted feedback to github that they should do so as well.

resolv_wrapper also isn't found in the current Travis CI (I was wrong
about that), because it's not in the Ubuntu package repository in 16.04.
 That's fixable if we move to 18.04.

I haven't worked through the Windows build issue yet.

I'm dissatisfied with the amount of repetition in build.yml, and will
see if I can find any good ways to reduce it.  We can of course create a
shell script like we did for Travis.
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev
Reply | Threaded
Open this post in threaded view
|

Re: Use GitHub Actions for builds

Michael Mattioli
I sorta kinda got macOS to work. You can see the build log here:
https://pipelines.actions.githubusercontent.com/maPQCN5hQS1QALnR9eGA8bjyfQToKa7Rzo1o1MArufvRc8nQZL/_apis/pipelines/1/runs/30/signedlogcontent/6?urlExpires=2019-12-02T23%3A58%3A13.2192164Z&urlSigningMethod=HMACV1&urlSignature=WpUdDNoZLUsf4MRxymzD9WM99NCc9elIU6oD9VbwI7w%3D

If you scroll to the bottom you'll see there seems to be a compile error.

I also had to add automake to the brew instal list for autoreconf.

For the Linux builds, I was able to define a matrix so that the commands to
setup, build, etc. are re-used. I also added the DEBIAN_FRONTEND env
variable; I previously tried setting this through the .yml file but adding
it with sudo seemed to work better. The last piece I'm trying to figure out
is how to set environment variables on a per-job basis when using a matrix.
At the end, when it does the git ls-file check it always exits with exit
code 1...

Check out this particular run to see all of the builds:
https://github.com/mmattioli/krb5/runs/330195951

Mike


On Mon, Dec 2, 2019 at 5:35 PM Greg Hudson <[hidden email]> wrote:

> On 12/2/19 6:12 AM, Michael Mattioli wrote:
> > What compiler, flags, etc. did you want to test on macOS? They only
> provide the latest version of macOS (10.15, Catalina). I can give it a spin
> and see if it works.
>
> clang, and the same flags as the first Ubuntu clang job.  I tried to set
> this up with Travis a couple of years ago and ran into two issues:
>
> 1. It was too slow--generally about 15 minutes to run a build and check
> instead of 7-8.  Github Actions uses the same macOS hosting provider as
> Travis (Macstadium), so might have the same problem.  But it's worth a try.
>
> 2. The macOS images have System Integrity Protection enabled, which
> interferes with the test suite.  The most significant problem is that
> the shell clears DYLD_LIBRARY_PATH from the environment.  There also
> appear to be some signal restrictions which break the iprop tests.  My
> attempts to work around these problems got as far as
>
> https://github.com/greghudson/krb5/commit/f99f4e9c79372516a8905e840e9ddc696c97c81c
> but the make-host-keytab.plin change is broken; it needs to suppress
> output and ignore errors when running one of the kadmin ank commands,
> without using the shell, and I couldn't find an easy way to do that in
> Perl.  That's a surmountable problem, but I got frustrated with the slow
> builds and stopped at that point.  It's possible that Github Actions
> turns off SIP on its macOS runners; otherwise we'll need the same
> workarounds.
>
> [I wrote:]
> >> while the Travis integration currently doesn't skip anything.  I will
> >> look into why slapd can't be installed and why resolv_wrapper isn't
> found.
>
> slapd can be installed if we do "sudo DEBIAN_FRONTEND=noninteractive
> apt-get ...".  Travis arranges for the noninteractive front-end by
> default.  I submitted feedback to github that they should do so as well.
>
> resolv_wrapper also isn't found in the current Travis CI (I was wrong
> about that), because it's not in the Ubuntu package repository in 16.04.
>  That's fixable if we move to 18.04.
>
> I haven't worked through the Windows build issue yet.
>
> I'm dissatisfied with the amount of repetition in build.yml, and will
> see if I can find any good ways to reduce it.  We can of course create a
> shell script like we did for Travis.
>
_______________________________________________
krbdev mailing list             [hidden email]
https://mailman.mit.edu/mailman/listinfo/krbdev