thread safety of sync_client::publish?

Hello,
when trying to embed the very easy to use sync_client into my software I came to a point where I asked myself if it is safe to publish information from different threads using the sync client.
First I tried to search the documentation for thread safety but wasn't able to find anything. After that I studied parts of your code and noticed that you use a mutex to lock some of the endpoint's data structures. On the other side you use boost::asio::write which is only thread safe for messages until some specified size (maybe the size is big enough for most mqtt messages). So I assume that the usage of sync_client::publish is thread safe (until messages are greater than the specified size) but I am a bit afraid to just try it out because multithreading errors are sometimes pretty hard to spot. And tbh I only read some small parts of the code - sometimes important points are also hard to spot.
Could you please give me an answer if you would consider it safe to use the sync_client's publish function concurrently in multiple threads on Windows and Linux?
Thank you in advance.

short answer
Unsafe
Analysis
sync_client's publish is processed the following part of code:
https://github.com/redboltz/mqtt_cpp/blob/v4.0.0/include/mqtt/endpoint.hpp#L9962-L9970
    // Blocking write
    template <typename MessageVariant>
    void do_sync_write(MessageVariant&& mv) {
        boost::system::error_code ec;
        if (!connected_) return;
        if (h_pre_send_) h_pre_send_();
        write(*socket_, const_buffer_sequence<PacketIdBytes>(std::forward<MessageVariant>(mv)), ec);
        if (ec) handle_error(ec);
    }

And if the endpoint is a simple TCP connection, then call following function:
https://github.com/redboltz/mqtt_cpp/blob/v4.0.0/include/mqtt/tcp_endpoint.hpp#L74-L77
    template <typename... Args>
    std::size_t write(Args&& ... args) {
        return as::write(tcp_, std::forward<Args>(args)...);
    }

        return as::write(tcp_, std::forward<Args>(args)...);

is https://www.boost.org/doc/libs/1_70_0/doc/html/boost_asio/reference/write/overload1.html.
And it finally calls https://www.boost.org/doc/libs/1_70_0/doc/html/boost_asio/reference/basic_stream_socket/write_some/overload1.html internally.
It is a member function of basic_stream_socket.
The bottom of the documents said that

Thread Safety
Distinct objects: Safe.
Shared objects: Unsafe.

The case you asking is shared object. So it is unsafe.
I wrote the PoC code.
Running demo: https://wandbox.org/permlink/MoKx94YtPqmUqcCW
// clang++ -std=c++17 client_server_multi.cpp -pthread  -ggdb3 -fsanitize=address -fno-omit-frame-pointer
#include <iostream>
#include <thread>
#include <vector>
#include <random>

#include <boost/asio.hpp>
#include <boost/lexical_cast.hpp>

int main() {
    // Common Config
    constexpr std::uint16_t port = 12345;
    std::random_device rd;
    std::mt19937 mt(rd());
    std::uniform_int_distribution<int> dist(1, 10);

    constexpr std::size_t const num_of_alphabet = 26;
    constexpr std::size_t const repeat = 5;

    constexpr std::size_t const publish_threads = 10;
    constexpr std::size_t const client_ios_threads = 4;

    // Server
    std::thread th_s([&] {
        boost::asio::io_service ioss;

        boost::asio::ip::tcp::acceptor ac(
            ioss,
            boost::asio::ip::tcp::endpoint(boost::asio::ip::tcp::v4(), port)
        );
        boost::asio::ip::tcp::socket ss(ioss);
        std::function<void()> do_accept;
        std::function<void(boost::asio::ip::tcp::socket&)> do_read;
        std::vector<boost::asio::ip::tcp::socket> connections;

        do_read =
            [&](auto& s) {
                auto buf = std::make_shared<std::string>();
                buf->resize(num_of_alphabet * repeat);
                async_read(
                    s,
                    boost::asio::buffer(*buf, num_of_alphabet * repeat),
                    [&, buf] (boost::system::error_code const& e, std::size_t /*rs*/) {
                        if (e) {
                            std::cout << e.message() << std::endl;
                        }
                        else {
                            std::cout << *buf << std::endl;
                            do_read(s);
                        }
                    }
                );
            };
        do_accept =
            [&] {
                ac.async_accept(
                    ss,
                    [&]
                    (boost::system::error_code const& e) {
                        if (e) {
                            std::cout << e.message() << std::endl;
                        }
                        else {
                            connections.emplace_back(std::move(ss));
                            auto& s = connections.back();
                            do_read(s);
                        }
                    }
                );
            };

        do_accept();
        ioss.run();
    });

    // Client
    std::function<void(boost::asio::ip::tcp::socket&)> do_send;
    do_send =
        [&](auto& s) {
            std::vector<std::thread> ths;
            ths.reserve(publish_threads);
            for (std::size_t i = 0; i != publish_threads; ++i) {
                ths.emplace_back(
                    [&] {
                        std::this_thread::sleep_for(std::chrono::milliseconds(dist(mt)));
                        std::vector<char> buf;
                        for (std::size_t i = 0; i != repeat; ++i) {
                            for (std::size_t j = 0; j != num_of_alphabet; ++j) {
                                buf.emplace_back('a' + j);
                            }
                        }
                        // send without lock or strand
                        boost::asio::write(s, boost::asio::buffer(buf));
                    }
                );
            }
            for (auto& t : ths) t.join();
        };

    std::thread th_c1(
        [&] {
            boost::asio::io_service iosc;
            boost::asio::strand strand(iosc.get_executor());
            boost::asio::ip::address address = boost::asio::ip::address::from_string("<IP_ADDRESS>");
            boost::asio::ip::tcp::endpoint endpoint(address, 12345);
            boost::asio::ip::tcp::socket sock(iosc);

            sock.async_connect(
                endpoint,
                [&]
                (boost::system::error_code const& ec) {
                    if (ec) {
                        std::cout << ec.message() << std::endl;
                    }
                    else {
                        std::cout << "client connected" << std::endl;
                        iosc.post(
                            [&] {
                                do_send(sock);
                            }
                        );
                    }
                }
            );
            std::vector<std::thread> v;
            v.reserve(client_ios_threads);
            for (std::size_t i = 0; i != client_ios_threads; ++i) {
                v.emplace_back(
                    [&] {
                        iosc.run();
                    }
                );
            }
            for (auto& t : v) t.join();
        }
    );

    th_c1.join();
    th_s.join();

}

It calls write() from multi threads. In my code, no problems were happened.
I guess that it work well by change. And we shouldn't depends on the behavior. It won't work well on other platforms or implementation updates in the future.
I tried to find the way to execute write() synchronously using strand. But I couldn't find it.
For example, I tried to update
                        // send without lock or strand
                        boost::asio::write(s, boost::asio::buffer(buf));

with
                        boost::asio::dispatch(strand, [&] {
                            return boost::asio::write(s, boost::asio::buffer(buf));
                        });
                        // I expect `buf` have been written into the socket at this point.

.
But it doesn't work.
Workaround
https://www.boost.org/doc/libs/1_70_0/doc/html/boost_asio/reference/io_context.html
is thread safe. See

Thread Safety
Distinct objects: Safe.
Shared objects: Safe, with the specific exceptions of the restart() and notify_fork() functions. Calling restart() while there are unfinished run(), run_one(), run_for(), run_until(), poll() or poll_one() calls results in undefined behaviour. The notify_fork() function should not be called while any io_context function, or

So you can post the lambda expression that contains publish() call to the io_context using https://www.boost.org/doc/libs/1_70_0/doc/html/boost_asio/reference/io_context/post.html.
NOTE: I know the function is deprecated but mqtt_cpp uses old style. We will update in the future.
By the way, if you post publish() into io_context, there is still a way to send it parallel.
Here is the part of the code:
            std::vector<std::thread> v;
            v.reserve(client_ios_threads);
            for (std::size_t i = 0; i != client_ios_threads; ++i) {
                v.emplace_back(
                    [&] {
                        iosc.run();
                    }
                );
            }
            for (auto& t : v) t.join();

Calling io_context::run() from multi  threads.

Thank you very much for your detailed answer.
The idea to use io_context::post seems very interesting and easy to implement.
Tbh I did not think of using it before (I am not very experienced with boost::asio). Therefore I already used a mutex. I don't expect high lock contention. I guess this is safe as long as no callback/handler will try to lock my mutex when an inner mutex is locked (so both mutexes will be locked once outer mutex first, inner mutex second and another time inner first, outer second)?

Even if multi threads write to the same socket, it is serialized in the socket buffer internally (it happens underlying layer of mqtt_cpp). So there is no performance advantage.
I guess that the reason of multi threading is application processing. In this case, posting io_service (io_context) is good choice.
If you satisfied my answer, please close the issue.

Thank you.


Progress Ring Chart

I'd like to define a color for each data on Progress Ring, it's possible?

I would like to give each legend on a Progress Chart a different color.
How can this be done?

Hi did anyone find a resolution for this? This is a very fine library with everything in place except that am stuck with this one issue before production of my application. Any help would be appreciated! Thanks in advance.

const data = {
    data: [0.4, 0.2, .09, 0.8],
    colors: ["red", "blue", "green", "yellow"],
  };

Obviously, use hex colors but this is the way to do it.

const data = {
    data: [0.4, 0.2, .09, 0.8],
    colors: ["red", "blue", "green", "yellow"],
  };

Obviously, use hex colors but this is the way to do it.

const data = {
    data: [0.4, 0.2, .09, 0.8],
    colors: ["red", "blue", "green", "yellow"],
  };

Obviously, use hex colors but this is the way to do it.

Hey! Tried this and it won't change the colors at all, any advice on integrating this with the color component in chartConfig?

@avnikakkar  Hey there, if you are still struggling use this example export default function ProgressRing() {   const screenWidth = Dimensions.get('window').width;   const {colors} = useTheme();   const data: ProgressChartData = {     labels: ['Web servers', 'Operating systems', 'Programming languages' , 'JavaScript' , 'Web Frameworks'], // optional     data: [0.2, 0.6, 0.8 , 0.1 , 0.25],     colors: [       'rgba(255, 0, 0,0.5)',       'rgba(238, 130, 238,0.6)',       'rgba(106, 90, 205,0.5)',       'rgba(60, 179, 113,0.2)',       'rgba(255, 172, 71 , 0.3)'     ],   };   return (     <View>       <ProgressChart         data={data}         width={Dimensions.get('window').width - 10}         height={170}         strokeWidth={7}         hasLegend={true}         withCustomBarColorFromData={true}         radius={35}         chartConfig={{           backgroundGradientFromOpacity: 0.5,           backgroundGradientToOpacity: 1,           backgroundColor: colors.surface,           backgroundGradientFrom: colors.surface,           backgroundGradientTo: colors.surface,           propsForLabels:{             fill:colors.text           },           decimalPlaces: 2,           color: (opacity = 1, _index) => rgba(0,0,0,${opacity}),         }}         style={{           marginVertical: 8,           borderRadius:10,         }}       />     </View>   ); }

export default function ProgressRing() {   const screenWidth = Dimensions.get('window').width;   const {colors} = useTheme();   const data: ProgressChartData = {     labels: ['Web servers', 'Operating systems', 'Programming languages' , 'JavaScript' , 'Web Frameworks'], // optional     data: [0.2, 0.6, 0.8 , 0.1 , 0.25],     colors: [       'rgba(255, 0, 0,0.5)',       'rgba(238, 130, 238,0.6)',       'rgba(106, 90, 205,0.5)',       'rgba(60, 179, 113,0.2)',       'rgba(255, 172, 71 , 0.3)'     ],   };   return (     <View>       <ProgressChart         data={data}         width={Dimensions.get('window').width - 10}         height={170}         strokeWidth={7}         hasLegend={true}         withCustomBarColorFromData={true}         radius={35}         chartConfig={{           backgroundGradientFromOpacity: 0.5,           backgroundGradientToOpacity: 1,           backgroundColor: colors.surface,           backgroundGradientFrom: colors.surface,           backgroundGradientTo: colors.surface,           propsForLabels:{             fill:colors.text           },           decimalPlaces: 2,           color: (opacity = 1, _index) => rgba(0,0,0,${opacity}),         }}         style={{           marginVertical: 8,           borderRadius:10,         }}       />     </View>   ); }

@avnikakkar Hey there, if you are still struggling use this example

that format was killing me...

export default function ProgressRing() {
  const screenWidth = Dimensions.get("window").width;
  const { colors } = useTheme();
  const data = {
    labels: [
      "Web servers",
      "Operating systems",
      "Programming languages",
      "JavaScript",
      "Web Frameworks",
    ], // optional
    data: [0.2, 0.6, 0.8, 0.1, 0.25],
    colors: [
      "rgba(255, 0, 0,0.5)",
      "rgba(238, 130, 238,0.6)",
      "rgba(106, 90, 205,0.5)",
      "rgba(60, 179, 113,0.2)",
      "rgba(255, 172, 71 , 0.3)",
    ],
  };
  return (
    <View>
      <ProgressChart
        data={data}
        width={Dimensions.get("window").width - 10}
        height={170}
        strokeWidth={7}
        hasLegend={true}
        withCustomBarColorFromData={true}
        radius={35}
        chartConfig={{
          backgroundGradientFromOpacity: 0.5,
          backgroundGradientToOpacity: 1,
          backgroundColor: colors.surface,
          backgroundGradientFrom: colors.surface,
          backgroundGradientTo: colors.surface,
          propsForLabels: { fill: colors.text },
          decimalPlaces: 2,
          color: (opacity = 1, _index) => `rgba(0,0,0,${opacity})`,
        }}
        style={{ marginVertical: 8, borderRadius: 10 }}
      />
    </View>
  );
}

@avnikakkar, I no longer have an example to share. I switched from react-native-chart-kit to react-native-svg-charts.

You can define a colors array and use the index

how can we use different off colors for different bars?

Hey there,
Can we set a shadow around each Progress Ring?

can we displace the legends itself


YARN-11738 Modernize SecretManager config

Description of PR
Make hash algorithm at SecretManager configurable.

hadoop.security.hmac-algorithm: The name of the hashing algorithm. Default: HmacSHA1
hadoop.security.hmac-length: The length of the random keys to use. Default: 64

How was this patch tested?

unit test modified
deployed and run mapred PI example, while i modified the yarn-site.xml and mapred-site.xml with HmacSHA256 and 1024 key length

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
20m 44s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
44m 53s

trunk passed

+1 :green_heart:
compile
20m  4s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
18m 46s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
1m 18s

trunk passed

+1 :green_heart:
mvnsite
1m 50s

trunk passed

+1 :green_heart:
javadoc
1m 23s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 56s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 43s

trunk passed

+1 :green_heart:
shadedclient
38m 23s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  3s

the patch passed

+1 :green_heart:
compile
19m 16s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
19m 16s

the patch passed

+1 :green_heart:
compile
19m 10s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
19m 10s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 13s

the patch passed

+1 :green_heart:
mvnsite
1m 46s

the patch passed

+1 :green_heart:
javadoc
1m 16s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 58s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
2m 47s

the patch passed

+1 :green_heart:
shadedclient
38m 17s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 57s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
1m  5s

The patch does not generate ASF License warnings.

258m 35s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 627da8f5e004 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / dfef74ce331b7448f14831654d1354aa4d5dc31c

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/1/testReport/

Max. process+thread count
3148 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  1s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
16m 39s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
33m  4s

trunk passed

+1 :green_heart:
compile
20m 15s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
19m  3s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
4m 24s

trunk passed

+1 :green_heart:
mvnsite
2m 52s

trunk passed

+1 :green_heart:
javadoc
2m 20s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m 53s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
4m 16s

trunk passed

+1 :green_heart:
shadedclient
36m 50s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 32s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 39s

the patch passed

+1 :green_heart:
compile
19m 20s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
19m 20s

the patch passed

+1 :green_heart:
compile
18m 47s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
18m 47s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
4m 17s

the patch passed

+1 :green_heart:
mvnsite
2m 48s

the patch passed

+1 :green_heart:
javadoc
2m 14s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
1m 49s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
4m 42s

the patch passed

+1 :green_heart:
shadedclient
37m 42s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
20m 54s

hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 19s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  2s

The patch does not generate ASF License warnings.

289m 33s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux e87184f7ce14 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 79738f820bca3f2cb601326440a6fccdee1cd826

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/2/testReport/

Max. process+thread count
3151 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@K0K0V0K , thanks for working on this!
If we change the HMAC and key length, then the keys generated before this change and after this change won't be the same.  Then, this become an incompatible change.
We probably need to add confs to set the algorithm and key length for backward compatibility.

Thanks @K0K0V0K for working on this. I agree with @szetszwo that this probably should be configurable.

Hi @szetszwo and @brumi1024 !
First of all thanks for the review. I modified the PR, now it is configurable. I now to create a configuration object at class static method is a bit risky, but that was the quick win what i found.
May i ask a review again? Thanks!

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
20m 59s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m  2s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
33m 12s

trunk passed

+1 :green_heart:
compile
20m 29s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
19m  4s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
4m 20s

trunk passed

+1 :green_heart:
mvnsite
2m 48s

trunk passed

+1 :green_heart:
javadoc
2m 19s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 51s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 18s

trunk passed

+1 :green_heart:
shadedclient
37m 26s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 32s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 34s

the patch passed

+1 :green_heart:
compile
19m 16s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
19m 16s

the patch passed

+1 :green_heart:
compile
18m 44s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
18m 44s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
4m 24s

the patch passed

+1 :green_heart:
mvnsite
2m 49s

the patch passed

+1 :green_heart:
javadoc
2m 13s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 48s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 42s

the patch passed

+1 :green_heart:
shadedclient
37m 29s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
20m 49s

hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 19s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  2s

The patch does not generate ASF License warnings.

308m  2s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 2bb1aac2b091 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9854cee29c8ddd3f68c8abb5c96587904b87315d

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/3/testReport/

Max. process+thread count
3151 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 20s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 33s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m 21s

trunk passed

+1 :green_heart:
compile
9m  0s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
8m 15s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
2m  9s

trunk passed

+1 :green_heart:
mvnsite
1m 34s

trunk passed

+1 :green_heart:
javadoc
1m 27s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 13s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
2m 26s

trunk passed

+1 :green_heart:
shadedclient
21m 23s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 52s

the patch passed

+1 :green_heart:
compile
8m 31s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
8m 31s

the patch passed

+1 :green_heart:
compile
9m  8s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
9m  8s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
2m 37s

the patch passed

+1 :green_heart:
mvnsite
1m 27s

the patch passed

+1 :green_heart:
javadoc
1m  7s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m  1s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
2m 38s

the patch passed

+1 :green_heart:
shadedclient
22m 36s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 43s

hadoop-common in the patch passed.

+1 :green_heart:
unit
23m  5s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
0m 35s

The patch does not generate ASF License warnings.

174m 21s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 6a2100d6fd97 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 349566210dc41ad2c0b755d97c8884d86fc1604a

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/5/testReport/

Max. process+thread count
3155 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 52s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
33m  5s

trunk passed

+1 :green_heart:
compile
20m  2s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
19m 18s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
4m 33s

trunk passed

+1 :green_heart:
mvnsite
2m 48s

trunk passed

+1 :green_heart:
javadoc
2m 20s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 54s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 24s

trunk passed

+1 :green_heart:
shadedclient
37m 15s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 33s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 37s

the patch passed

+1 :green_heart:
compile
19m 23s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
19m 23s

the patch passed

+1 :green_heart:
compile
18m 47s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
18m 47s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
4m 15s

the patch passed

+1 :green_heart:
mvnsite
2m 47s

the patch passed

+1 :green_heart:
javadoc
2m 15s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 48s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 41s

the patch passed

+1 :green_heart:
shadedclient
37m 51s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
20m 49s
/patch-unit-hadoop-common-project_hadoop-common.txt
hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 22s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  1s

The patch does not generate ASF License warnings.

288m 13s

Reason
Tests

Failed junit tests
hadoop.conf.TestCommonConfigurationFields

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux f3ffc5c5b384 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d23e2e9bb83459891c0786b1a9dad7fa68080670

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/6/testReport/

Max. process+thread count
1255 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m  9s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
33m  2s

trunk passed

+1 :green_heart:
compile
20m  9s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
19m  7s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
4m 25s

trunk passed

+1 :green_heart:
mvnsite
2m 47s

trunk passed

+1 :green_heart:
javadoc
2m 18s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 52s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 20s

trunk passed

+1 :green_heart:
shadedclient
37m 27s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 33s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 35s

the patch passed

+1 :green_heart:
compile
19m 44s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
19m 44s

the patch passed

+1 :green_heart:
compile
19m 17s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
19m 17s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
4m 22s
/results-checkstyle-root.txt
root: The patch generated 4 new + 110 unchanged - 0 fixed = 114 total (was 110)

+1 :green_heart:
mvnsite
2m 50s

the patch passed

+1 :green_heart:
javadoc
2m 15s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 53s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 39s

the patch passed

+1 :green_heart:
shadedclient
38m 34s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
20m 59s

hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 23s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  4s

The patch does not generate ASF License warnings.

290m 22s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/7/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux c8167cf2c945 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ea7f701931d197038b15725fb81b0a46f8369899

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/7/testReport/

Max. process+thread count
1348 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/7/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hi @brumi1024 @szetszwo!
May i ask a review again? Thanks!

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 49s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 51s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
33m 44s

trunk passed

+1 :green_heart:
compile
28m 12s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
24m 44s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
4m 51s

trunk passed

+1 :green_heart:
mvnsite
2m 40s

trunk passed

+1 :green_heart:
javadoc
2m 12s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 45s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 31s

trunk passed

+1 :green_heart:
shadedclient
38m 42s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 30s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 37s

the patch passed

+1 :green_heart:
compile
21m 36s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
21m 36s

the patch passed

+1 :green_heart:
compile
22m 38s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
22m 38s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
4m 37s
/results-checkstyle-root.txt
root: The patch generated 7 new + 110 unchanged - 0 fixed = 117 total (was 110)

+1 :green_heart:
mvnsite
2m 39s

the patch passed

+1 :green_heart:
javadoc
2m  8s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 46s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 44s

the patch passed

+1 :green_heart:
shadedclient
38m 51s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
21m  4s

hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 30s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  3s

The patch does not generate ASF License warnings.

311m  8s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/9/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 197e792f927e 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 38759d63b0e0b2f1967e79046425fdd0835c025c

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/9/testReport/

Max. process+thread count
1890 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/9/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 58s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 57s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
34m  3s

trunk passed

+1 :green_heart:
compile
29m 49s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
26m 19s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
4m 51s

trunk passed

+1 :green_heart:
mvnsite
2m 48s

trunk passed

+1 :green_heart:
javadoc
2m 12s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 44s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 21s

trunk passed

+1 :green_heart:
shadedclient
38m 49s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 31s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 51s

the patch passed

+1 :green_heart:
compile
23m 31s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
23m 31s

the patch passed

+1 :green_heart:
compile
22m 36s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
22m 36s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
4m 48s
/results-checkstyle-root.txt
root: The patch generated 7 new + 110 unchanged - 0 fixed = 117 total (was 110)

+1 :green_heart:
mvnsite
2m 47s

the patch passed

+1 :green_heart:
javadoc
2m  7s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 46s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 42s

the patch passed

+1 :green_heart:
shadedclient
38m 58s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
21m 17s

hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 26s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  3s

The patch does not generate ASF License warnings.

318m 39s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 365391e4fb7a 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 38759d63b0e0b2f1967e79046425fdd0835c025c

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/8/testReport/

Max. process+thread count
1256 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/8/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 49s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 53s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
34m 21s

trunk passed

+1 :green_heart:
compile
30m 18s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
23m 19s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
4m 47s

trunk passed

+1 :green_heart:
mvnsite
2m 45s

trunk passed

+1 :green_heart:
javadoc
2m  8s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 44s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 30s

trunk passed

+1 :green_heart:
shadedclient
39m 13s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 31s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 32s

the patch passed

+1 :green_heart:
compile
22m 51s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
22m 51s

the patch passed

+1 :green_heart:
compile
22m 13s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
22m 13s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
4m 54s
/results-checkstyle-root.txt
root: The patch generated 7 new + 110 unchanged - 0 fixed = 117 total (was 110)

+1 :green_heart:
mvnsite
2m 39s

the patch passed

+1 :green_heart:
javadoc
2m  8s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 41s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 52s

the patch passed

+1 :green_heart:
shadedclient
38m 33s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
21m  9s

hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 32s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  3s

The patch does not generate ASF License warnings.

314m 44s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/10/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 14eb9963779a 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 6d41273269933e813a6ce9d8880d149970c79aca

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/10/testReport/

Max. process+thread count
2448 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/10/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 56s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 57s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
34m 36s

trunk passed

+1 :green_heart:
compile
31m 55s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
23m 21s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
4m 47s

trunk passed

+1 :green_heart:
mvnsite
2m 45s

trunk passed

+1 :green_heart:
javadoc
2m 11s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 44s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 31s

trunk passed

+1 :green_heart:
shadedclient
39m  6s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 31s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 39s

the patch passed

+1 :green_heart:
compile
23m 18s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
23m 18s

the patch passed

+1 :green_heart:
compile
23m 37s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
23m 37s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
4m 53s
/results-checkstyle-root.txt
root: The patch generated 7 new + 110 unchanged - 0 fixed = 117 total (was 110)

+1 :green_heart:
mvnsite
2m 41s

the patch passed

+1 :green_heart:
javadoc
2m  6s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 46s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
4m 46s

the patch passed

+1 :green_heart:
shadedclient
38m 24s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
21m 14s

hadoop-common in the patch passed.

+1 :green_heart:
unit
27m 29s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
1m  4s

The patch does not generate ASF License warnings.

317m 27s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/11/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 6db9bd7b3386 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 6d41273269933e813a6ce9d8880d149970c79aca

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/11/testReport/

Max. process+thread count
3152 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/11/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 18s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 11s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m 18s

trunk passed

+1 :green_heart:
compile
8m 50s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
9m 13s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
checkstyle
2m 31s

trunk passed

+1 :green_heart:
mvnsite
1m 34s

trunk passed

+1 :green_heart:
javadoc
1m 16s

trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 59s

trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
2m 24s

trunk passed

+1 :green_heart:
shadedclient
22m 24s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 22s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 52s

the patch passed

+1 :green_heart:
compile
8m 34s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
8m 34s

the patch passed

+1 :green_heart:
compile
8m 26s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
javac
8m 26s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
2m  5s
/results-checkstyle-root.txt
root: The patch generated 7 new + 110 unchanged - 0 fixed = 117 total (was 110)

+1 :green_heart:
mvnsite
1m 30s

the patch passed

+1 :green_heart:
javadoc
1m 21s

the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
1m 14s

the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

+1 :green_heart:
spotbugs
2m 40s

the patch passed

+1 :green_heart:
shadedclient
21m 11s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 51s

hadoop-common in the patch passed.

+1 :green_heart:
unit
23m 28s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
0m 40s

The patch does not generate ASF License warnings.

174m 33s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/12/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7144

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint

uname
Linux 871d226e3d63 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 75a062c435cfe384f365fb96770fbcd3ae8c0217

Default Java
Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/12/testReport/

Max. process+thread count
3154 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7144/12/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


terragrunt hclfmt should not evaluate interpolations

It sound like terragrunt hclfmt is trying to evaluate interpolations or even using AWS creds.
I'm getting the following error :
[terragrunt] 2019/08/08 15:08:51 Reading Terragrunt config file at /path/to/root/terragrunt.hcl
[terragrunt] 2019/08/08 15:08:56 /path/to/root/terragrunt.hcl:26,34-57: Error in function call; Call to function "find_in_parent_folders" failed: ParentFileNotFound: Could not find a entity.tfvars in any of the parent folders of /path/to/root/terragrunt.hcl. Cause: Traversed all the way to the root.., and 7 other diagnostic(s)
[terragrunt] 2019/08/08 15:08:56 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

Here is my repo structure :
├── root
├── environment
│   ├── live-module-1
│   │   └── terragrunt.hcl
│   ├── live-module-2
│   │   └── terragrunt.hcl
└── terragrunt.hcl

Versions

Terraform v0.12.6
Terragrunt v0.19.19

Good point! In fact, hclfmt shouldn't do any decoding since it doesn't need to! Opened https://github.com/gruntwork-io/terragrunt/pull/826 as a potential fix.

Fixed in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.19.22. Binaries will show up shortly


bypass PageCache for compact_level0_phase1

### Tasks
- [ ] https://github.com/neondatabase/neon/pull/8543
- [ ] https://github.com/neondatabase/aws/pull/1666
- [x] create staging log scraping alert for the warnings added: https://neonprod.grafana.net/alerting/grafana/cdtf6y667hatcc/view
- [x] create prod log scraping alert for the warnings added: https://neonprod.grafana.net/alerting/grafana/edtf7wry2qzggb/view
- [ ] https://github.com/neondatabase/aws/pull/1663
- [ ] https://github.com/neondatabase/aws/pull/1724
- [ ] https://github.com/neondatabase/aws/pull/1743
- [ ] https://github.com/neondatabase/infra/pull/1745
- [ ] https://github.com/neondatabase/neon/pull/8769

The compact_level0_phase1 currently uses ValueRef::load here, which internally uses read_blob with the FileBlockReader against the delta layer's VirtualFiles. This still goes through the PageCache for the data pages.
(We do use vectored get for create_image_layers, which also happens during compaction. But I missed the compact_level0_phase1.)
Complete PageCache Bypass
We can extend the load_keys step here to also load the lengths of each blob into memory (instead of just the offset)
https://github.com/neondatabase/neon/blob/9b98823d615c991422b6edd3ec3197192f763cf2/pageserver/src/tenant/timeline/compaction.rs#L498-L503
This allows us to go directly to the VirtualFile when we use the ValueRef here:
https://github.com/neondatabase/neon/blob/9b98823d615c991422b6edd3ec3197192f763cf2/pageserver/src/tenant/timeline/compaction.rs#L623
The problem with this: we'd lose the hypothetical benefits of PageCache'ing the data block if multiple ValueRefs are on the same page.
Do we rely on the PageCache for performance in this case?
Yes, production shows we do have >80% hit rate for compaction, even on very busy pageservers.
One instance by example:

Quick Fix 1: RequestContext-scoped mini page cache.
In earlier experiments, I used a RequestContext-scoped mini page cache for this.
Problem with this is that if more layers need to be compacted than we have pages in the page cache, it will start thrashing.
Proper Fix
Use streaming compaction with iterators where each iterator caches the current block.
We do have the diskbtree async stream now.
We could wrap that stream to provide a cache for the last-read block.

Did some initial scouting work on this:

the holes functionality of compaction (introduced in https://github.com/neondatabase/neon/pull/3597) requires scanning of all keys before scanning all values

tl;dr for what holes does:

So the trade-off is that we rather create smaller L1s than creating sparse delta space on top of image layers.
And if we're ingesting a lot of data on both sides of the hole, then this is definitely the right trade-off because we will have full-sized L1s on either side.

More details in private Slack DM

we need to preserve the holes functionality until we have a better approach for image layer creation at the top
testing: I can't find any dedicated Rust unit tests. Would be nice to extract the existing logic enough to get coverage for existing behaviors, but that's a lot of work.

Status update:

Initial PR merged #8543

Read the PR description, it contains a detailed rollout strategy.

default neon_local, unit tests, regression tests, and Staging are running a validation mode

log scraping alerts for staging and prod are configured
these will detect validation failures and forward to #on-call-storage-{staging,prod}-stream

post-merge of above PR, I ran some manual tests to inspect overhead of validation mode

results here: https://github.com/neondatabase/neon/pull/8543#issuecomment-2260710812

pre-prod and prod require the following PR to be merged before the release next week:

https://github.com/neondatabase/aws/pull/1663
If we don't merge it, pre-prod and prod will run in validating mode as well, which we probably don't want yet.

Status update:

No validation failures in Staging
Spot-checked flaky tests dashboard & error messages, no indication that the validation failure is happening there, either.

https://github.com/neondatabase/cloud/issues/16404 would be better than spot-checking

Plan / needs decision:

Validation mode in pre-prod region? Measure CPU impact, see if low enough so we can enable validation mdoe in prod. Probablistic validation?

Status update: validation mode enabled in pre-prod
First night's prodlike cloudbench run had concurrent activity from another benchmark, smearing results: https://neondb.slack.com/archives/C06K38EB05D/p1723797560693199
However, here's the list of dashboards I looked at:

Cloudbench metrics
4GS
Node Exporter Full
CPU
VirtualFile
/storage disk IOPS
Memory
Fleet overview => memory usage
Time spent doing th ecompactions, scraped from logs

Preliminary interpretation (compare time range from 0:00 to 8:00, that's where the load happens)

no noticable CPU / disk IOPS impact
but compaction iterations take about 2x wall clock time

makes sense because validation does about twice the amount of VirtualFile calls, with no concurrency
we're not bottlenecking on disk however

What's the practical impact? 2x wall-clock-time-slower compactions means double the wait time on the global semaphore for compactions (assuming that semaphore is the practical throughput bottleneck, which I believe is the case). In other teams, it means we only achieve half the usual compaction throughput.
So, is prod compaction throughput bottlenecked on the global semaphore?
We can use the following query to approximate business of the semaphore (%age of tenants waiting for permit):
(pageserver_background_loop_semaphore_wait_start_count{instance="pageserver-8.eu-west-1.aws.neon.build",task="compaction"} - pageserver_background_loop_semaphore_wait_finish_count)
/on(instance) pageserver_tenant_states_count{state="Active"}

There are some places where we have sampling skew, so, do clamping
clamp(
(pageserver_background_loop_semaphore_wait_start_count{task="compaction"} - pageserver_background_loop_semaphore_wait_finish_count)
/on(instance) sum by (instance) (pageserver_tenant_states_count)
, 0, 1)

the p99.9 instance in that plot looks like this
quantile(0.999,
clamp(
(pageserver_background_loop_semaphore_wait_start_count{task="compaction"} - pageserver_background_loop_semaphore_wait_finish_count)
/on(instance) sum by (instance) (pageserver_tenant_states_count)
, 0, 1)
)

average like this
avg(
clamp(
(pageserver_background_loop_semaphore_wait_start_count{task="compaction"} - pageserver_background_loop_semaphore_wait_finish_count)
/on(instance) sum by (instance) (pageserver_tenant_states_count)
, 0, 1)
)

For posterity, there was a Slack thread discussing these results / next steps: https://neondb.slack.com/archives/C033RQ5SPDH/p1723810312846849

Decision from today's sync meeting:

https://github.com/neondatabase/infra/pull/1745
Create metric to measure semaphore contention.
Table decision for remaining regions until EOW / next week.

This week, as per discussion thread:

analyze perf impact in pre-prod (enable new mode, without validation)
no changes to prod

Results from pre-prod are looking good.

cloudbench results for that time range
Less wall clock time for the same workload (see screenshot below) => GOOD
unchanged memory for that time range unchange
unchanged CPU for that time range

Plan:

Roll the non-validating mode into more prod regions this week.

Results from rollout shared in this Slack thread
tl;dr:

halved the PS PageCache eviction rate, and stabilized it a lot
halved the metric "wall clock time spent on compaction / ingested bytes" (see query below)

sum by (neon_region) (rate(pageserver_storage_operations_seconds_global_sum{operation="compact",neon_region=~"$neon_region"}[$__rate_interval]))
/
sum by (neon_region) (rate(pageserver_wal_ingest_bytes_received[$__rate_interval] / 1e6))


Separate Agents and Optimizer

Optimizers should not be initialized inside the Agents.

Closing this for now, as a plan for massive overhaul of the core is in order. :)


consider markerScale for donut and chart markers

This will address https://github.com/VEuPathDB/web-monorepo/issues/301 and more, which considers markerScale.
Changes are: a) arc width of Donut marker; b) sum label of Donut marker; c) sum label margin of Chart marker; d) sum label of Chart marker. Screenshots before and after this PR.

before this PR

after this PR

screenshots with this PR based on https://github.com/VEuPathDB/web-monorepo/pull/285

Thank you for your reviews and tests, @bobular 👍


Data class named Subscription causes schema to show a subscription

Describe the bug
I have a data model class called Subscription but no GraphQL subscriptions setup yet. Hot Chocolate 11.2.2 shows a /Subscription schema item (the GraphQL subscription) with the properties of my data class.
To Reproduce
Steps to reproduce the behavior:

Setup empty Hot Chocolate API
Create a data class called Subscription
Setup a root Query to it:

public IQueryable<Subscription> GetSubscriptions([ScopedService] MyDbContext context)
            => context.Subscriptions.AsNoTracking();

Run the project, reload schema in Banana Cake Pop
Observe both the /Query/subscriptions and the /Subscription paths are populated.

Expected behavior
No /Subscription schema should be set up unless I explicitly do so.
Desktop:
OS: Windows 10
<PackageReference Include="HotChocolate.AspNetCore" Version="11.2.2" />
<PackageReference Include="HotChocolate.Data" Version="11.2.2" />

5.0.201

Additional context

public class Subscription
{
    public Guid SubscriptionId { get; set; }
    public DateTime Created { get; set; }
    public User User { get; set; } = default!;
    public Account Account { get; set; } = default!;
    public IList<Connector> Connectors { get; set; } = new List<Connector>();
}

This happens because we infer that class as a subscription root. Which is wrong in your case :)
I will put it to the backlog.

Thanks for reporting!

I have looked at the issue for this and we decided to not fix it for 12.
There is a workaround for this available:
services
    .AddGraphQL()
    .ModifyOptions(o => SubscriptionTypeName = "N_123456")

By setting the subscription type name to something does not exist it will not pick the default name Subscription and thus such a type in the schema would not become a root type.
Rest assured we will fix this, we just decided that this issue is one we can move one more time since the impact is big.


Generalize VivViewer to Take a List of configurations (for overlays)

Right now VivViewer only displays one layer at a time.  We should change the API to accept multiple layers with potentially multiple loaders.

Now that VivViewer takes multiple views, we should move towards an API that takes in lists of configurations in order - views, layers, sliders, colors etc.  This is out of scope of #139 but is essential work that will make this whole thing sing.


Filter for item in lists

There should be an autocomplete or filter feature to search for a specific entity in a list
child of #35

This issue was moved to BlueBrain/nexus#157


Added backend for stock details page

http://<IP_ADDRESS>:5000/stock/detail?stock=AAPL
changing stock variable at end of URL changes which stock to query.  Frontend not implemented, so currently just shows name and exchange

Don't forget to add the TD number to your pull request.

I'm approving the pull request, but you need to make a correction before merging.
The @login_required has to be between the @route decorator and before the function def. Otherwise the @login_required wont work.
Make the decorators look like the following:

Once you fix that, you should be good to go to merge the branch.


Not working on Xcode 7.2.1

Hey @ftronnbe, I just tested on my machine with 7.2.1 (7C1002) and it works fine.
Did you get asked to load additional bundles and pressed NO?
This SO thread handles a lot of gotchas that can be encountered with plugins, so check it out and please let me know if any of it works.
http://stackoverflow.com/questions/30361228/why-are-my-xcode-plugins-such-as-clang-format-installed-with-alcatraz-no-longe

Namely, try this defaults read com.apple.dt.Xcode DVTPlugInManagerNonApplePlugIns-Xcode-7.2.1 and see if Peckham is in the allowed list.

Thanks @markohlebar! I suspected a problem with Peckham since all my other plugins were working correctly after the Xcode update. I tried to reinstall the plugin using alcatraz but to no avail.
I then re-installed alcatraz and then re-installed Peckham. It is now working correctly.

@ftronnbe wow, that's strange that reinstalling both worked. Thanks for getting back with this, I'll add some troubleshooting section for posterity.


Update on API Documentation

Some editorial improvements on API documentation — differ between reference definition and user manual documentation.

👍

:+1:


dnvm upgrade. Determining latest version Error

Hello,
I followed the instructions to install dnx on macosx and when I do :   dnvm upgrade
or other comands available, it doesn't work. What ever I do, shows:
$  dnvm upgrade
Determining latest version
Error: Could not find latest version from feed https://www.nuget.org/api/v2
I followed others solutions that I read in others issues, i tried all, but doesn't work anything.
I tried doing with http feed  --> http://www.nuget.org/api/v2:
dnvm upgrade DNX_FEED=http://www.nuget.org/api/v2
but shows other error after doing some things:
HTTP Error  fetching dnx-mono.1.0.0-beta6 from http://www.nuget.org/api/v2
Why it this happen? It can be fixed?

Can you log this in https://github.com/aspnet/dnvm ? Thanks!

Ok! Thanks! --> https://github.com/aspnet/dnvm/issues/382


Frame params

[x] send_frame API
[x] keyframe placement logic
[ ] capi/crav1e to expose it? (@dwbuiten do you want to give it a go?)

(It sits on top of the API cleanup)

Coverage increased (+0.007%) to 81.214% when pulling eb91b4630147c5410bdd0c4fcbffe3e79f7da274 on rust-av:frame-params into 7f1e8837329758810fcb5c7e6f5d799272f3e0cd on xiph:master.

Looks easy enough to expose.

Now the feature is implemented and tested.

I think this shouldn't conflict with #1481. compute_lookahead_data is self contained, so moving the call to that into send_frame shouldn't cause any issues.

Discussing with @tdaede this takes the priority over the scenechange improvements since looks like it needs further iterations and testing.
I'd land it today and wait for the other issues pending in the project to be completed.


What is the "bd" folder?

Doesn't seem like it's documented anywhere. Would be awesome if it was stored in bd/README.md or similar

Just kidding, it's already in the main README, my bad.


enforceEmailVerification and case-sensitive

I have this scenario:

i have set enforceEmailVerification and sendVerificationEmail to True
user sign up from his computer with mail<EMAIL_ADDRESS>and complete registration using the confirmation mail
everything is working using the mail<EMAIL_ADDRESS>user try to login from his phone, but his smartphone makes the first letter uppercase, so the user type<EMAIL_ADDRESS>user get the error "mail not verified"

Setting enforceEmailVerification to false i have no problem at all, but i want the user to confirm his mail. I'm not sure if i'm doing something wrong or if this is an actual bug, any idea on how to fix it?

I get this exact same problem.  I suspect the problem is with this code in core/lib/server.js starting at line 125:
      var ok = true;
      var loginEmail = attempt.methodArguments[0].user.email;
      if (loginEmail) {
        var email = _.filter(user.emails, function(obj) {
          return obj.address === loginEmail;   // THIS NEEDS TO BE A CASE INSENSITIVE COMPARISON
        });
        if (!email.length || !email[0].verified) {
          ok = false;
        }
      }

I don't know how to replace useraccounts:core with my own version to test this though.  Can anyone help?


PATCH request failure

Are there any examples of a patch request in practice? I may have the incorrect syntax but when I try something like (for some video_id of mine): v.patch('https://api.vimeo.com/videos/video_id', {"data":{"privacy":{"view":"anybody"}}}) I get a 200 response but it doesn't actually change the privacy setting. Is this a bug or am I doing something wrong?

Try dropping the data key, and just sending the request body directly
{"privacy":{"view":"anybody"}}

Same issue. I get a 200 but the privacy doesn't change.

@hate5six the Vimeo library uses requests under the hood. So the correct way to send a requests with query parameters shown in this example using our library would be:
v.patch('/videos/{video_id}', data={"privacy":{"view":"anybody"}})


Custom logging condition (clean history)

This is a follow-up of #59:

As promised, here is a PR trying to address an issue that many developers may encounter: how to provide my own condition for logging?
What I propose here is quite simple:

The test that checked if the request is in logging_methods is moved to its own method _should_log. This method is called at the end of the process, before returning response, so that we can provide both request and response to this method. This way, we could implement any logic we want in it.
Rename LoggingMixin to BaseLoggingMixin so that developers can easily extend it with their own _should_log method. LoggingMixin just inherits BaseLoggingMixin without any changes.
To address more specifically issue #52, propose a LoggingErrorsMixin which will only log errors by implementing a _should_log method that log only responses with status_code >= 400.

If you need more details about the implementation, feel free to ask.
If we merge this, we'll have to update the README to explain these new features.

This PR is the same but with a clean Git history, as discussed with @avelis.
Regards 🙂

@frankie567 Sorry I just saw this now.

@frankie567 Thanks again for the much needed enhancement! Really appreciate it! And I also appreciate the diligence to re-open the PR from #59 to preserve the git commit history. I can't express the thanks on that one.

@avelis Don't worry, I know we're not working on the same timezone 😉
It was nothing really, just a few cherry-pick and that was it! Glad you appreciate my proposals!


Spike: Investigate publishing different versions of our components

Info
Once we release the new Angular wrappers we will have to support two separate versions of react-components, and angular-components. Very likely supporting the current angular-components won't be an issue, but there is a high likelihood of having to make updates to the two different versions of react-components. We need to be setup and ready to publish both a version 5 and a version 6 of react-components
Acceptance Criteria:

What is the effect semantic-release has on our release process?
Do we need to update our Github actions for this?
Do we need to update our branching strategy (ie. set up a new branch for React v5)?
Can we use dry-run to test our release strategy to make sure we can release both versions?

:tada: This issue has been resolved in version 5.4.1 :tada:
The release is available on:

npm package (@lts dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This issue has been resolved in version 1.29.1 :tada:
The release is available on:

npm package (@lts dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Expose V5 APIs

🚨 Snapshot command failed
Snapshots are updated automatically in following commit https://github.com/superdispatch/web-ui/pull/426/commits/44399ea5f7e46a6b7865e7472746121b3a2be09e
Please review before merging.

🚨 Snapshot command failed
Snapshots are updated automatically in following commit https://github.com/superdispatch/web-ui/pull/426/commits/8aa016ac45ec0bfb7bf684773577914eb77b315b
Please review before merging.

🚨 Snapshot command failed
Snapshots are updated automatically in following commit https://github.com/superdispatch/web-ui/pull/426/commits/39f114dbe3477428d005f440d0395971aa13bf4f
Please review before merging.


How do I remove Ignored Files and Directory Paths from Project Settings

I mistakenly added a path from a directory outside of the workspace. Then after clicking "Ignore Above Path" and adding it to the list, I realized the mistake and tried to delete it. If I remove it the list in the Ignored Paths textarea, it just repopulates the next time I try to add a different path. Nothings gets deleted. Only added to.

Hi @kbanse. Thanks for letting me know. I believe this might be a bug with with Nova not properly saving when the UI element loses focus. I will report this to Panic and see if they can shed any light on the problem, but I sense this will have to be patched in Nova itself.
In the meantime, when you make a change to a text box try clicking into another text box before leaving the preferences. I am usually able to get the preference to save this way. Hope this helps!

A quick update. Panic confirmed that this is a bug in Nova and is something they will work on patching in the next release.

I understand this is still an issue in Nova 1.4. I will update Panic and continue to hope this can be addressed in a future release.

I understand this is still an issue in Nova 1.4. I will update Panic and continue to hope this can be addressed in a future release.

This issue still hasn't been fixed as of Nova 5.1. It has been some time since I reported the issue, so I will reach out to Panic again see if they can take another look.

I have contacted Panic again for updates. Will update with any new information.

This was resolved in the version 3.0 release by using different preference UI options.


Add some buttons / options to change the color of images.

Cool idea! Go for it!

thank you sir