Incorect results on big data set(11ml records)

Hello,
I'm trying to run cosinesimil_sparse space with hnsw methods on 11 millions records dataset.
And when i try to fuzzy match company name i have 60/40 results, i 100% confident that vector consist company.
Then i create index, and run knnQueryBatch by k=30, and in 60 percent times in top 30 isn't needed company(company names are not similar at all)
in 40% times algorithm find correct company.
My create index parameters: are {
"M": 16,
"indexThreadQty": 24,
"efConstruction": 100,
"post": 0,
}
i create tfidf by TfidfVectorizer from sklearn.
i try m = 16, efConstruction = 512 - bad results
m = 20, efConstruction = 100,200 - bad results
m = 100, efConstruction = 100,200 - bad results(
Thanks for reply)

Hi @DavidRimskiy this is not an error per se, there may be multiple reasons why it doesn't perform well:

possible duplicates in the data set
low value of the parameter ef.
As a sanity check you can try using an inverted index (https://github.com/nmslib/nmslib/blob/master/manual/methods.md) : https://github.com/nmslib/nmslib/blob/master/manual/methods.md

It can be slowish, but results will be exact.

Hi @DavidRimskiy this is not an error per se, there may be multiple reasons why it doesn't perform well:

possible duplicates in the data set
low value of the parameter ef.
As a sanity check you can try using an inverted index (https://github.com/nmslib/nmslib/blob/master/manual/methods.md) : https://github.com/nmslib/nmslib/blob/master/manual/methods.md

It can be slowish, but results will be exact.

Thanks for answear)
I've tried before inverted index (space:negdotprod_sparse_fast method:simple_ividix)
but without custom create index parameters and results was bad.
Vector did not consist duplicates, and entire dataframe too, i drop duplicates each time before run script.
By your answear i need to low efConstruction to 50 or lower, or replace it by efSearch = 50?

Does bad mean slow or inaccurate? Normally (modulo some possible bugs) it should be exact matching. Don't forget that the highest cosine similarity doesn't necessarily mean you get a result you want. In particular, negdotprod_sparse_fast is the maximum inner product search. So the highest result don't indicate the best match per se: you can have entries with multiple repeating words and they will have a higher score. Whether results are good or bad first needs to be compared with the brute-force search. If HNSW has a similar recall, it's the problem with encoding and the similarity metric rather than with k-NN search per se.
With respect to parameter selection, reasonable values are M=30, efConstruction=200, efSearch in the range of 50 to 2000. Oftentimes, it needs to be pretty high.

Does bad mean slow or inaccurate? Normally (modulo some possible bugs) for the inverted index it should be exact matching.
Don't forget that the highest similarity doesn't necessarily mean you get a result you want. In particular, negdotprod_sparse_fast is the maximum inner product search. So the highest result don't indicate the best match per se: you can have entries with multiple repeating words and they will have a higher score. Whether results are good or bad first needs to be compared with the brute-force search. If HNSW has a similar recall, it's the problem with encoding and the similarity metric rather than with k-NN search per se.
With respect to parameter selection, reasonable values are M=30, efConstruction=200, efSearch in the range of 50 to 2000. Oftentimes, it needs to be pretty high.

Thanks for Answer)
bad mean inaccurate)
I will try with your suggestions and let you know.

Does bad mean slow or inaccurate? Normally (modulo some possible bugs) for the inverted index it should be exact matching.
Don't forget that the highest similarity doesn't necessarily mean you get a result you want. In particular, negdotprod_sparse_fast is the maximum inner product search. So the highest result don't indicate the best match per se: you can have entries with multiple repeating words and they will have a higher score. Whether results are good or bad first needs to be compared with the brute-force search. If HNSW has a similar recall, it's the problem with encoding and the similarity metric rather than with k-NN search per se.
With respect to parameter selection, reasonable values are M=30, efConstruction=200, efSearch in the range of 50 to 2000. Oftentimes, it needs to be pretty high.

Hi, i tried your suggest approach, and the result on 11ml records
cosinesimil_sparse space with hnsw method:
{M = 30, efConstruction = 200, ef = 2000} = inaccurate
{M = 30, efConstruction = 200, ef = 1000} = inaccurate
{M = 20, efConstruction = 200, ef = 200} = inaccurate
negdotprod_sparse_fast = inacurate(
but i tried cosinesimil_sparse space with hnsw method locally 1Ram 4cpu on 1 mln record
with {M = 16, efConstruction = 100, ef = 100} without ef and result was inaccurate, but with ef = 100 result was perfect)

Great!


Core.MMSNavigationSample - Create MMS Site Collection Group

I see that the Core.MMSNavigationSample code skips over site collection groups. I need to create a group that has IsSiteCollectionGroup set to true. Is this even possible with CSOM?

Nevermind.  I figured it out, I just didn't see the PnP code doing this at all.  The answer is to call the GetSiteCollectionGroup on the TermStore object.  Seems obvious, but I just didn't look at the API closely enough.


./configure: error:

Please can you explain to me why I keep getting this error when I run ./install.sh on terminal and how can I fix it?
/configure: error: the HTTP gzip module requires the zlib library.
You can either disable the module by using --without-http_gzip_module
option, or install the zlib library into the system, or build the zlib library
statically from the source with nginx by using --with-zlib= option.
ERROR: failed to run command: sh ./configure --prefix=/etc/nginx/nginx ...
[-] Failed to configure openresty installation.
I will really appreciate your quick response. Thanks in anticipation.

+1 Same problem

you need zlib development bindings:
apt-get install --reinstall zlibc zlib1g zlib1g-dev

Hello @jackanders62, Evilginx only support Debian right now but you can help us in the experiment docker image here: https://github.com/kgretzky/evilginx/pull/45
You should able to run this on any operating system with Docker installed.

above tips worked for me

https://www.cyberciti.biz/faq/centos-install-zlib-devel/
yum install zlib-devel


chore(FDS-401): [Danger.js] - Validate if target branch requires release branch

Validate if the target branch requires the release branch.
PR Author Checklist

[ ] PR title adheres to Conventional Commits
[ ] I have finished reviewing the contents of this PR for accuracy
[ ] If needed, I have included a Changeset and it follows Semantic Versioning principles
[ ] I have completed all of the above and have enabled auto-merge for this PR

Fails

:no_entry_sign:
Do not target main directly from develop. A separate release branch is needed

  Generated by :no_entry_sign: dangerJS against 8b428fc42e65620e25b0036d7dae7b4a1009b1da


🐛: Search Functionality not working

:red_circle: Title :  Fix Search Functionality
:red_circle: Bug :  The search functionality is not working. When a user tries to search for items, the system does not return any results or displays incorrect results.
:red_circle: Changes :   Capturing search input and applying logic using JavaScript.
Screenshots 📷
https://github.com/Avdhesh-Varshney/WebMasterLog/assets/83902558/c469e49d-a58d-4593-9312-e3b7463b776b

:white_check_mark: To be Mentioned while taking the issue :

Full name :  Vaishali Ajmera
GitHub Profile Link : https://github.com/Vaishali-ajmera
What is your participant role?  GSSOC contributor

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎

Thank you for the opportunity. Please add gssoc and label tag to this issue

@Vaishali-ajmera without assigning the issue, never raise the pr.


Fix array parameter

Doxygen xml has an optional 'array' in parameters, append this to parameters on output.
Fixes #184.

thank you for the report and fix!  I am getting things ready to get a new release out and want to get this in there.  i will be adding the test case you have provided to c_maths project, could you please rebase this PR in the meantime?  (sorry, it is late, but I definitely want to get you a valid release for your project!!)

I've just rebased this, thanks for looking into it!
Ralf


22.05: nixops fails with "waiting for exclusive access to the Nix store for ca drvs "

After updating a system to NixOS 22.05 beta via nixops, further update attempts (while the machine runs 22.05) lock up with:
machine......> waiting for exclusive access to the Nix store for ca drvs...

and stay there forever.
Rebooting the machine into 21.11 and redeploying works around the issue.

To exhibit the bug, experimental-features =  ca-derivations must be enabled systemwide on the target machine. Another workaround is to disable it.

Looks like a nixops bug, https://github.com/NixOS/nixops/issues/1520


s/DataSource/Datasource for consistency with quasar

Long overdue. This changes the manifest attribute from DataSource-Module to be Datasource-Module, so this is technically breaking. However, the launcher/frontend should be unaffected since they do their own thing and don't use assembleDatasource.

@alissapajer done! :)


Introduce Build-owned isolated models

@bot-gradle test qfl

I've triggered the following builds for you. Click here to see all build failures.

Quick Feedback - Linux Only (Trigger) (Check)


brazil_states.geojson

Hi,'brazil_states.geojson' can not use,showing blank in jupyter notebook.

There appears to be nothing wrong with the data file and it functions perfectly in the app.
If you can provide more information on the issue, feel free to reopen this.


[FEATURE] Build images on Mac OSX (with solution)

Building on Mac OSX
Description
I have tried to build your images on Mac OSX which failed as awaited, BUT it is pretty easy to solve and I was able to build everything locally. 👍
The problem with the original shell script and Mac OSX is that Apple has BSD grep preinstalled in contrast to GNU grep. On Mac OSX one can simply install GNU grep with Homebrew:
brew install grep

Afterwards, one can set grep to the PATH variable, while this might change some system behaviour. I have changed locally all grep calls in the script with ggrep and it all worked. This might be possible to be interchangeable by setting the command as variable (e.g. in build.yml?). Then the build script should work on Linux and Mac OSX.

Hi @alexeyegorov, the solution proposed on #42 looks pretty simple indeed! I am figuring out how to test on my Linux machine and I will let you know. Any ideas?

Thanks for your reply. :)
I can't help you with that. I can only admit that it works for me on my MacBook Pro 2019. Unfortunately, I don't know how you could test it on Linux. I can run Linux on MacOS, but I don't think it is possible the other way around. I record my screen and share it with you to prove that it is working. ;)

Thanks for your reply. :)
I can't help you with that. I can only admit that it works for me on my MacBook Pro 2019. Unfortunately, I don't know how you could test it on Linux. I can run Linux on MacOS, but I don't think it is possible the other way around. I record my screen and share it with you to prove that it is working. ;)

An idea that came into my mind: is it possible to install BSD version of grep on Linux? This was at least one of the things that wasn't working on Mac as it has BSD version preinstalled.

An idea that came into my mind: is it possible to install BSD version of grep on Linux? This was at least one of the things that wasn't working on Mac as it has BSD version preinstalled.


API Changes: Sol-arena

Rename function
In sol_arena_slice_dup_str_n() and sol_arena_slice_dup_str() rename
the str argument to src
In sol_arena_slice_dup() rename the slice argument to src
issue #1659
Signed-off-by: Amanda Coelho<EMAIL_ADDRESS>

have you changed anything from last sol-arena pr?

merged.


Unable to integrate with Winston logger

Issue
We're unable to get applicationinsights to communicate with our AppInsights instance via winston.
Versions

Tool
Version

Node
20.12.0

NPM
10.5.2

applicationinsights
3.0.1

winston
3.13.0

Steps to reporduce
I've attached a minimum reproduceable example here:
applicationinsights-winston-issue.zip
In the example, we currently have the following package.json file:
{
  "name": "applicationinsights",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "applicationinsights": "^3.0.1",
    "winston": "^3.13.0"
  }
}

And the following index.js file:
let appInsights = require('applicationinsights');

appInsights.setup('<MY INSTRUMENTATION KEY>')
  .setAutoCollectRequests(true)
  .setAutoCollectPerformance(true, true)
  .setAutoCollectExceptions(true)
  .setAutoCollectDependencies(true)
  .setAutoCollectConsole(true, true)
  .setAutoCollectPreAggregatedMetrics(true)
  .setSendLiveMetrics(false)
  .setInternalLogging(false, true)
  .enableWebInstrumentation(false)
  .start()

const { createLogger, transports } = require('winston');

const logger = createLogger({
  transports: [new transports.Console()]
})

logger.info('Info Test');
logger.warn('Warn Test');
logger.error('Error Test');

Install dependencies with npm i
Run the file with npm t
See the errors and logs in the terminal:npm t

><EMAIL_ADDRESS>test
> node index.js

@azure/opentelemetry-instrumentation-azure-sdk [
  'Module @azure/core-tracing has been loaded before @azure/opentelemetry-instrumentation-azure-sdk so it might not work,
please initialize it before requiring @azure/core-tracing'
]
Invalid metric name: "\ASP.NET Applications(??APP_W3SVC_PROC??)\Request Execution Time". The metric name should be a ASCII string with a length no greater than 255 characters. []
Invalid metric name: "\ASP.NET Applications(??APP_W3SVC_PROC??)\Requests/Sec". The metric name should be a ASCII string with a length no greater than 255 characters. []
Invalid metric name: "\Process(??APP_WIN32_PROC??)\Private Bytes". The metric name should be a ASCII string with a length no greater than 255 characters. []
Invalid metric name: "\Memory\Available Bytes". The metric name should be a ASCII string with a length no greater than 255 characters. []
Invalid metric name: "\Processor(_Total)\% Processor Time". The metric name should be a ASCII string with a length no greater than 255 characters. []
Invalid metric name: "\Process(??APP_WIN32_PROC??)\% Processor Time". The metric name should be a ASCII string with a length no greater than 255 characters. []
Accessing resource attributes before async attributes settled []
Extended metrics are no longer supported. Please reference the Azure Monitor OpenTelemetry Migration Doc for more information. If this functionality is required, please revert to Application Insights 2.X SDK. []
{"level":"info","message":"Info Test"}
Accessing resource attributes before async attributes settled []
{"level":"warn","message":"Warn Test"}
Accessing resource attributes before async attributes settled []
{"level":"error","message":"Error Test"}
Accessing resource attributes before async attributes settled []
Accessing resource attributes before async attributes settled []
Accessing resource attributes before async attributes settled []

See that no logs are sent to Azure AppInsights

What else we've tried
We've also tried manually sending the logs to AppInsights with:
const { TelemetryClient } = require('applicationinsights');
const { Writable } = require('stream');

const appInsights = new TelemetryClient('<MY INSTRUMENTATION KEY>');

const stream = new Writable({
  write: message => {
    appInsights.trackTrace({ message: message.toString() });
  }
});

const { createLogger, transports } = require('winston');

const logger = createLogger({
  transports: [new transports.Console(), new transports.Stream({ stream })]
})

logger.info('Info Test');
logger.warn('Warn Test');
logger.error('Error Test');

When running like this, then we get the following logs:

This appears to work, but then we get an additional info log (bottom of the above example image) and a few of the Accessing resource attributes before async attributes settled [] error logs (top and 2nd from the bottom of the above example image) as well.
This solution also requires us to install the stream package, which is not ideal.
How to fix?
We are receiving logs from our other applications, so AppInsights appears to be working, but we're unable to communicate via Winston from on NodeJS app.
How can we go about integrating with Winston?

@JPStrydom I was able to fix your example by modifying your index.js to be the following:
let appInsights = require('applicationinsights');

appInsights.setup("your-instrumentation-key")
  .setAutoCollectRequests(true)
  .setAutoCollectPerformance(true, true)
  .setAutoCollectExceptions(true)
  .setAutoCollectDependencies(true)
  .setAutoCollectConsole(true, true)
  .setAutoCollectPreAggregatedMetrics(true)
  .setSendLiveMetrics(false)
  .setInternalLogging(false, true)
  .enableWebInstrumentation(false)
  .start()

const { createLogger, transports } = require('winston');

const logger = createLogger({
  transports: [new transports.Console()]
})

async function test () {
  logger.info('Info Test');
  logger.warn('Warn Test');
  logger.error('Error Test');
  await appInsights.defaultClient.flush();
}

test();

The modification just ensures that the application waits to exit until the Application Insights SDK has gotten a chance to export the Winston logs. Please let me know if this example makes sense or if you have any other questions. Thanks!

Hi @JacksonWeber !
Thanks for taking the time to reach out. I tried your example but still don't receive any logs on our Azure App Insights instance.

Our use-case is to pass our Winston logs from our FeathersJS back-end to our Azure AppInsights.
We experienced the same behavior on our back-end, which is constantly running while the logs are being sent, so I'm not sure if flushing the client is the solution here.
Was your thinking that the client needs to be flushed after every log?

@JPStrydom It's great you were able to see the Winston logs! Those other logs you're seeing are warnings and they come from a few sources. The "accessing resource attributes" warning is from OpenTelemetry and reports this warning because we have a VM resource detector that makes an API call to a VM metadata service. Given that you're not running in that environment, the call times out, but using the SDK before that timeout generates this warning for now. It won't impact functionality for you.
The warning regarding extended metrics is because you're attempting to set a configuration option that used to modify the behavior of collection of extended metrics. However, like the warning mentions, we don't support extended metrics in versions greater than or equal to 3.0.0.

Hi @JacksonWeber
Thanks again!
Given the setup config we have:
  .setAutoCollectRequests(true)
  .setAutoCollectPerformance(true, true)
  .setAutoCollectExceptions(true)
  .setAutoCollectDependencies(true)
  .setAutoCollectConsole(true, true)
  .setAutoCollectPreAggregatedMetrics(true)
  .setSendLiveMetrics(false)
  .setInternalLogging(false, true)
  .enableWebInstrumentation(false)

Is there anything we need to change to get rid of the errors/warnings you mentioned?
I'd also just like some clarification on when exactly we need to call the appInsights.defaultClient.flush() function:

Do we need to call this after every log, OR
Do we only need to call this once, OR
Do we need to call it on an interval?

@JPStrydom For removing the extended metrics warning, simply change .setAutoCollectPerformance(true, true) to .setAutoCollectPerformance(true). You'll still auto collect performance counters, but the second parameter is for setting auto collection of extended metrics (which isn't supported).
As for the resource attributes warning, this isn't currently ignorable because we turn on resource detection by default for popular Azure Resources.
Regarding on when to use flush(). It entirely depends on your application. If Node.js process flow ends quickly after a call to log telemetry, then it may make sense to await a flush() however, in most cases, such as a long running Node.js server application there typically isn't any need to asynchronously await a flush.

@JPStrydom Please let me know if this has been able to resolve your issue, thanks!

@JPStrydom Please let me know if this has been able to resolve your issue, thanks!

We've temporarily paused development on the AppInsights integration, but as soon as we resume, I'll test and let you know. Thanks!


Bug: Select disappears and don't come back after accordion closes

When I have a select input inside an accordion, it only works if the accordion is open by default. As soon as it closes, it does not come back. Also, if the accordion is closed by default, it will never appear.
Here is a minimal example:
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8"/>
    <meta name="viewport" content="width=device-width"/>
    <title>Franken UI</title>
    <link rel="preconnect" href="https://rsms.me/"/>
    <link rel="stylesheet" href="https://rsms.me/inter/inter.css"/>

    <style>
        :root {
            font-family: Inter, sans-serif;
            font-feature-settings: "liga" 1, "calt" 1; /* fix for Chrome */
        }

        @supports (font-variation-settings: normal) {
            :root {
                font-family: InterVariable, sans-serif;
            }
        }
    </style>

    <!-- For stability in production, it's recommended that you hardcode the latest version in the CDN link. -->

    <link
            rel="stylesheet"
            href="https://unpkg.com/franken-ui/dist/css/core.min.css"
    />

    <script>
        const htmlElement = document.documentElement;

        if (
            localStorage.getItem("mode") === "dark" ||
            (!("mode" in localStorage) &&
                window.matchMedia("(prefers-color-scheme: dark)").matches)
        ) {
            htmlElement.classList.add("dark");
        } else {
            htmlElement.classList.remove("dark");
        }

        htmlElement.classList.add(
            localStorage.getItem("theme") || "uk-theme-zinc"
        );
    </script>

    <script
            type="module"
            src="https://unpkg.com/franken-ui/dist/js/core.iife.js"
    ></script>
    <script
            type="module"
            src="https://unpkg.com/franken-ui/dist/js/icon.iife.js"
    ></script>
</head>
<body class="bg-background text-foreground">
<!--  END -->
</body>
</html>

Initial render:

After closing and opening the accordion

When inspecting the HTML after  closing and opening the accordion, I only see:

Am I doing something wrong or is this a bug?

Hello @j0rd1smit
Yes this is a known re-rendering/reactivity issue. And thanks for bringing this up. I was not aware that it also affects accordion.

Not sure if it's related but uk-select does not work with modals either.


DO NOT MERGE Reverting the reversion of "Send topics to Publishing API and Rummager"

Reverts alphagov/hmrc-manuals-api#107
This PR adds the code added #105 again to send sub topics to the Content Store and Rummager. This is currently blocked from being merged until HMRC give the OK. Ticket.
Talk to myself, @jennyd, @mikejustdoit or @marksheldonGDS if you need more context.

Superseded by https://github.com/alphagov/hmrc-manuals-api/pull/110.


Make secret names user selectable

[ ] Stored on the client side
[ ] Optional: Make them exportable to TXT

[ ] CRUD options on secret name


socket close exception

when I send request to the server, "socket closed"  appeared quite often.
anyone else has knew this problem?

Nothing actionable to do here. You're better off trying to create a test case that demonstrates the problem, or asking on stackoverflow.


Scroll up position update

Successfully updated scroll up position to center.
#173 fixed

This branch has conflicts

add some screenshots to see your changes

@Suchitra-Sahoo Please review

Why your navbar is so disturbed. I have hust checked and everything is fine.


Getting an error when tryin to use geckolib

Here is the pastebin link:
https://pastebin.com/3vJ2T0b6

Get that geckolib BS out of here.


fix: resolved faulty spinner loading after error in console

Description

Added a try catch to stop spinner if Error is throw by generateModels() func.

Related issue(s)

Resolves #1607

Although this particular bug is fixed by adding try catch block ,  I am not very confident in the implementation and working of this command because i tried using interactive mode and it didn't work as it should .

@SahilDahekar can you attach some images of the error, and solution hard to understand what is happening.

@Souvikns
Before :
The Generating models spinner does not stop even after error.

In order to stop have to manually cancel the operation.

After :
No longer have faulty spinner and error logged to console.

/rtm


couldn't harvest

I had over 2000 XEM for more than 2000 blocks and NEM won't let me harvest.  I am not sure why.
This may or may not be related to the fact that I am running this NEM node on my Mac.  Interestingly enough nembex doesn't see my Mac even though I can send and receive XEM.  I think my router was configured for Windows so this might also be part of the problem.  I mention these things here because I am not sure if they are important or not.
http://s25.postimg.org/v9h5sk767/Screenshot_2015_02_13_12_04_10.png

@jabo38 right under balance, you have vested balance, that's the answer to your issue :)


[GOBBLIN-588] Remove final decoration to allow Azkaban password to be overwritten

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-588

Description

[x] Here are some details about my PR, including screenshots (if applicable):
1)Bump the version of Helix.
2)Remove the final decoration for AzkabanClient::password field to allow subclass to overwrite the value.
3) Increase the cluster integration test timeout from 1min to 2min in case of the Travis is slow.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@ibuenros can you review?


Adding more content to the compilation artifacts output

Component
Forge
Describe the feature you would like
Hi,
Running forge build leads to generating compilation artifacts in out with only abi/bytecode/deployed_bytecode. This limits the integration with other tools (like Slither, Echidna etc..).
Would it be possible to generate all the compilation artifacts? (or to have a flag for it)? The ones that are needed for crytic-tools are (and probably for most of the other third parties integration):

abi
ast
bin
bin-runtime
srcmap
srcmap-runtime
userdoc
devdoc
hashes [from 0.4.12]
compact-format [from 0.4.12, and until 0.8.10, after that this was dropped as the legacy ast format was removed]

I tried with --extra-output ast but it did not work. However because some of these flags depend on the compiler version, it would be great to let foundry figure out which ones are available.
Related: https://github.com/crytic/crytic-compile/issues/230
Additional context
No response

I think this should be possible via foundry toml? Cc @mattsse

this question was also asked in the support chat today by me and another individual.
I need to output the metadata.json so that I can verify on sourcify

@gakonst it would be great to have a flag's integration instead of the config file, as it would allow smooth third parties integration.
Otherwise, it would mean that every foundry's user that wants to use Slither need to update their config file (as opposed to a flag that Slither will directly use when running foundry)

Metadata is also needed for verification

better metadata bindings have been added to ethers-rs https://github.com/gakonst/ethers-rs/pull/894
Is there any (good) reason we should support configuring the output on a file or contract basis?
I feel only supporting  { "*": { "*": [<contract level settings>] } } is reasonable
this should be either a list of
output_selection = ["ir", "storageLayout", "evm",...]
or
output_selection = ["all"] which is { "*": { "*": [ "*" ], "": [ "*" ] } }
From the docs, the only file level config is ast, right?
what if we make a
metadata = true field to

set "metadata" field in output selection
write a metadata.json file for every contract

@sambacha ?

+1 on @montyly's point on also having a flag on the CLI to allow doing that without a Config file

better metadata bindings have been added to ethers-rs gakonst/ethers-rs#894
Is there any (good) reason we should support configuring the output on a file or contract basis? I feel only supporting { "*": { "*": [<contract level settings>] } } is reasonable
this should be either a list of
output_selection = ["ir", "storageLayout", "evm",...]
or output_selection = ["all"] which is { "*": { "*": [ "*" ], "": [ "*" ] } }
From the docs, the only file level config is ast, right?
what if we make a metadata = true field to

set "metadata" field in output selection
write a metadata.json file for every contract

@sambacha ?

The reason we would want this is for verification, which is dependent on how etherscan consumes this. I would guess that they normally get 1 metadata.json file, as opposed to 1 for every contract. I will email their support and ask, thanks @mattsse

@montyly should be fixed now! Will be released on today's nightly

I am not sure that @762 fixed this issue. I can't find a way to generate ast, or hashes (but I can generate all the others). Am I missing something?
For ast, it would be great if compact-format was enabled for solc from 0.4.12, and until 0.8.10.

wrt to hashes, do you evm.methodIdentifiers? https://docs.soliditylang.org/en/v0.4.12/using-the-compiler.html#input-description
The ast can only be set for the whole solc CompilerInput, it's enabled by default but we currently do move the ast to the Artifact.
will add this shortly.

Great, thanks.
For hashes: yes it is evm.methodIdentifiers; this one actually already works in forge with this name (I got confused because hashes is the naming with solc's --combined-json)
For the compact format, yes it is  --ast-compact-json (or compact-format through --combined-json)

just to be sure, the JSON interface returns contract metadata in case of an internal compiler error while the CLI interface does not for solc.

@montyly You can now get ast and evm.methodIdentifiers as well - good to close? :smile:

I think so - here's a slither job example https://github.com/thirdweb-dev/contracts/blob/main/.github/workflows/slither.yml#L27-L43. feel free to re-open if still an issue


fix contractor journal sanity check race condition

This PR fixes a rare, nondeterministic race in the renter's contractor where the sanity check would sometimes fail. Instead of reloading the journal from disk and sanity checking against the loading data, the contractor now has a lock-protected data contractorPersist field that is continuously updated under lock and used for the sanity check.

don't we want it to be reloading from disk though? I thought that was the purpose of the sanity check -- to confirm that the data on disk matches the data in memory.

Yeah lol the point was to make sure that the data on disk matched the data in memory.

I think the real issue here is that the contractor lock is grabbed before the journal lock, but the journal lock needs to be grabbed first to ensure data correctness

Worth noting that the contract manager has only one lock, we might want to replicate that. Actions affecting the log also affect the status of the manager correctness, so you make sure the manager cannot change while the log does and vice versa

Ah, that makes sense. I'll amend this commit to fix the locking patterns instead


Navigation not working with Tabs

See http://plnkr.co/edit/75V0tqqHOZHRwvm552rA
When clicking "navigate to Page2" the Tabs are overlayed by Page2. Clicking then "Navigate to Tab2" does not work at all. Same Problem on real Android Device.
When enabling Device Mode (use Ipad for space) in Chrome Desktop and reload plunker (F5), it starts working as expected. Also works on real Ios-Device, so seems MD-related

Hello! Thanks for opening an issue with us. Because we follow the material design standards, by default tabs are hidden by new pages on Android devices. Therefore, while it is actually navigating to the correct tab, you cannot actually see it because the tabs are hidden. If you would like for tabs on Android to not be hidden by new pages you can set tabSubPages to false in your ionicBootstrap.  I have made a plunker that shows how to do this and gives a demonstration of how it works. Thanks for using Ionic!

Thats great thanks! But why does it behave differently in Device-Mode?

@waybackrider Is tabSubPages not working for you on your Android device?

Yes it is! I meant Device-Mode in Chrome Desktop, there it also works without tabSubPages: false

@waybackrider Sorry if im misunderstanding but do you mean device mode emulating an ios device, such as an ipad, in chrome?

@jgw96 yes, but now i see this uses ios theme, therefore it works as expected! Thank you!

No problem!


The display of file path is delayed by data preview.

System Information (please complete the following information):

Model Builder Version (available in Manage Extensions dialog): <IP_ADDRESS>2802
Microsoft Visual Studio Enterprise 2019: 16.11.5

Describe the bug

On which step of the process did you run into an issue: Data page
Clear description of the problem: Browse to input a file data set, then find that the file path is displayed only after loaded the data preview.

To Reproduce
Steps to reproduce the behavior:

Select Create a new project from the Visual Studio 2019 start window;
Choose the C# Console App (.NET Core) project template with .Net 5.0;
Add model builder by right click on the project;
Select one scenario support file dataset;
On data page, click "Browse..." to input the data set;
See that the file path is displayed only after loaded the data preview.

Expected behavior
the file path is displayed after input the dataset, should not be affected by the data preview.
Screenshots
If applicable, add screenshots to help explain your problem.

This is lower priority. Adding to the data tab p1/p2 issue and closing for now.


Mangopay user updation not working.

I am using mango pay python adk version 3 to integrate mangopay. But when I trying to update a mangopay user I got the below error,
{APIError}{"Message":"One or several required parameters are missing or incorrect. An incorrect resource ID also raises this kind of error.","Type":"param_error","Id":"a58db418-69f7-4950-9bb0-8e6b01a98a41#1602149066","Date":1602149067.0,"errors":{"Nationality":"Object reference not set to an instance of an object.","CountryOfResidence":"Object reference not set to an instance of an object."}}
This is my code:
nationality_iso_code = convert_to_iso_country_code(request_data['country'])
country_of_residence_iso_code = convert_to_iso_country_code(request_data['country_of_residence'])
dob_in_utc = convert_date_to_utc(request_data['dob'])

natural_user = NaturalUser.get(user_profile.mango_pay_user_id)
natural_user.first_name = request_data['first_name']
natural_user.last_name = request_data['last_name']
natural_user.birthday = dob_in_utc,
natural_user.nationality = nationality_iso_code,
natural_user.country_of_residence = country_of_residence_iso_code,
natural_user.address.address_line_1 = request_data['address']['address_line_one']
natural_user.address.address_line_2 = request_data['address']['address_line_two']
natural_user.address.city = request_data['address']['city']
natural_user.address.postal_code = request_data['address']['post_code']
natural_user.address.country = nationality_iso_code
natural_user.save()

I don't Know Why I got this.

Looking into it 👍
Max

Hi there @backend-stunntech,
Please make sure that your convert_to_iso_country_code  and country_of_residence_iso_code  methods are returning strings.
The only way i managed to reproduce your issue was by setting those fields to objects:
user.nationality = {}
user.country_of_residence = {}

, which is not correct.
You should have something like this:
user.nationality = 'FR'
user.country_of_residence = 'FR'

Hi @backend-stunntech
Please let me know if we need to reopen this issue


improve device notification support

change how we send events (event names are now sent in an event field, not a method field)
the daemon command no longer requires to be run from a project root; the daemon.start operation sets the appropriate cwd for the start command
send log messages back over the wire protocol

LGTM


nwm_client does not work in FIPS environments

The current version of nwm_client uses concurrent.futures.ProcessPoolExecutor which tracks processes using md5 which is disabled by FIPS. This leads to the client hanging and never returning with data in a FIPS environment.
Switching to an async systems like nwm_client_new or using fsspec will hopefully fix this issue. It's also possible that upgrading to Python 3.10+ may resolve the issue, as well.

This issue is resolved by upgrading to Python 3.9+.


dropped the first and last 4 volumes in data loading

The paper says to drop the first and last 4 volumes

We did this according to http://studyforrest.org/annotation_timing.html


fix(gha/bot): update README when creating develop=>release PR

Signed-off-by: Tiago Castro<EMAIL_ADDRESS>

bors merge

bors cancel

bors ping

bors merge

bors try

Bors is broken AGAIN. If the try passes I'll simply merge manually.

bors merge

bors merge

bors cancel

bors merge

bors cancel


Add array of objects setting type for packages

This would be useful for packages that accept an unknown amount of user-defined values; for example, a package that would take a list of file extensions and a command to associate with them. Example data structure:
[
  {'extension': 'php', 'command': 'php', 'args': ''}
  {'extension': 'psc', 'command': 'PapyrusCompiler.exe', 'args': ''}
  {'extension': 'billy', 'command': 'bob', 'args': 'joe'}
  {'extension': 'foo', 'command': 'bar', 'args': 'bat'}
]

This could be represented in the settings page by having each object be multiple side-by-side text fields (one for each property), along with a "remove entry" button to remove the object from the array. Additionally, there'd be an "add entry" button that inserts another row with the fields with their default values.

Was an interface for this ever added?  I’m wanting to add an array of values for one of my settings.  I think it could be as simple as a single line where you enter values separated by commas.


Allow functions of tokens in @Inject()

I don't think this will work, it might be hard to differentiate, and currently this kind of logic is burried deep inside typedi. If tokens are stored in a separate constants.ts file and imported from there, there should be no reason for this function.


Better ratelimit handling when talking to the Matrix server

Currently we recommend people to ensure the bot has ratelimits disabled to make sure things like invites to chatrooms work with minimum delay. This, while acceptable for a first release, should be improved and made more robust in a future release. See discussion for details.

MSC3059 and Synapse PR 9648 are slightly different and changes to Synapse will be required if MSC3059 lands.


warg key new fails opening platform secure storage

@fibonacci1729
error: failed to set signing key with name `default` of registry `http://localhost:8090`: Platform secure storage failure: zbus error: org.freedesktop.Secret.Error.IsLocked: Cannot create an item in a locked collection

After some investigation I believe this is related to running in a headless environment. Since that is likely to be somewhat common for this tool I think we'll need an alternative way to store keys.

Hi @lann
Any updates on this end?
Today, I installed warg, and when I attempt to set the key for my local repository at <IP_ADDRESS>:8090, I encounter the same error. Subsequently, I installed gnome-keyring using the command:
sudo apt install gnome-keyring
Today, I installed warg, and when I attempt to set the key for my local repository at <IP_ADDRESS>:8090, I encounter the same error. Subsequently, I installed gnome-keyring using the command:
warg key new <IP_ADDRESS>:8090
error: failed to set signing key with name `default` of registry `https://<IP_ADDRESS>:8090/`: Couldn't access platform secure storage: SS error: result not returned from SS API

Any idea on how to resolve this?

Hi @lann
Any updates on this end?
Today, I installed warg, and when I attempt to set the key for my local repository at <IP_ADDRESS>:8090, I encounter the same error. Subsequently, I installed gnome-keyring using the command:
sudo apt install gnome-keyring
After installing gnome-keyring, I re-executed the warg command, and now I am encountering the following error:
warg key new <IP_ADDRESS>:8090
error: failed to set signing key with name `default` of registry `https://<IP_ADDRESS>:8090/`: Couldn't access platform secure storage: SS error: result not returned from SS API

Hi @lann
Any updates on this end?
Today, I installed warg, and when I attempt to set the key for my local repository at <IP_ADDRESS>:8090, I encounter the same error. Subsequently, I installed gnome-keyring using the command:
sudo apt install gnome-keyring
After installing gnome-keyring, I re-executed the warg command, and now I am encountering the following error:
warg key new <IP_ADDRESS>:8090
error: failed to set signing key with name `default` of registry `https://<IP_ADDRESS>:8090/`: Couldn't access platform secure storage: SS error: result not returned from SS API

Btw, I'm running warg in a headless environment.

Hi @lann
Any updates on this end?
Today, I installed warg, and when I attempt to set the key for my local repository at <IP_ADDRESS>:8090, I encounter the same error. Subsequently, I installed gnome-keyring using the command:
sudo apt install gnome-keyring
After installing gnome-keyring, I re-executed the warg command, and now I am encountering the following error:
warg key new <IP_ADDRESS>:8090
error: failed to set signing key with name `default` of registry `https://<IP_ADDRESS>:8090/`: Couldn't access platform secure storage: SS error: result not returned from SS API

Btw, I'm running warg in a headless environment and I have installed warg server In-memory storage following guideline https://github.com/bytecodealliance/registry/blob/main/crates/server/README.md

Take a look at https://github.com/bytecodealliance/registry/discussions/237 for some relevant discussion about warg development. We are taking another look at private key management for publishers since it still such a pain.

Thx! @lann


WIP: Feature/cypress

Description of the Change

Closes #
Alternate Designs

Possible Drawbacks

Verification Process

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my change.
[ ] All new and existing tests passed.

Changelog Entry

Credits

Props @

Hi @ivanlopez 👋
I know this PR is just a WIP and you didn't ask for any review yet. I just stumbled over it and saw that you are using a 3rd party WP Cypress Test Utils library. Over the last year or so the Open Source Practice has actually been working on our own suite of test utils that you can find here: https://github.com/10up/cypress-wp-utils. I'd love to see whether that meets your needs and if not what feedback you may have.
Ideally, we should try to standardize our approach on one solution and since we maintain one I am biased towards that.
Thanks in advance :)

I see they've added some more methods since I originally started on the branch. I'll see if I can get my baseline test to pass while using the 10up version. Thanks for flagging.

Closing this since this branch is stale, and testing is being looked at elsewhere internally.

@darylldoyle I'm curious where we're looking at including base tests as its something we might be able to help utilize things like https://github.com/10up/cypress-wp-setup && https://github.com/10up/cypress-wp-utils


Integration test guideline for views not working on TFS build agents

The method GetProjectPath in the code example https://docs.microsoft.com/en-us/aspnet/core/mvc/controllers/testing?view=aspnetcore-2.1#accessing-views assumes a fixed folder structure.
On TFS (and probably VSTS too), the folder structure changes, the binaries are e.g. in C:\Agent_work\1\b, the source in C:\Agent_work\1\s.
So any attempt to find the wwwroot  path will fail as the test is executed from within the binaries directory and the relative path does not match anymore.
Please provide a better way to find the wwwroot path which also works on TFS build agents
My current workaround is a post build which does a  xcopy of the wwwroot to the targetdirectory.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b5cce372-24e2-3489-cb19-c85d934e5d3b
Version Independent ID: 9d12b19f-a94e-4df4-357b-3f3b93696eb6
Content: Test controller logic in ASP.NET Core
Content Source: aspnetcore/mvc/controllers/testing.md
Product: aspnet-core
GitHub Login: @ardalis
Microsoft Alias: riande

Did you try the new testing infrastructure? ... https://docs.microsoft.com/aspnet/core/test/integration-tests
The underlying logic is different for inferring the content root: https://docs.microsoft.com/aspnet/core/test/integration-tests#how-the-test-infrastructure-infers-the-app-content-root-path
I can't guarantee the behavior with TFS build agents, but you'd also have the UseSolutionRelativeContentRoot setting at your disposal that might help.
@Rick-Anderson @scottaddie @rachelappel We didn't discuss this topic ... YET. It would've come up tho for the sample update at some point.

No I haven't yet tried the new infrastructure with this specific product.
Looking at  UseSolutionRelativeContentRoot https://github.com/aspnet/Hosting/blob/f9d145887773e0c650e66165e0c61886153bcc0b/src/Microsoft.AspNetCore.TestHost/WebHostBuilderExtensions.cs#L61, it's basically GetProjectPath just with a different name.
Looking at the new WebApplicationFactoryContentRootAttribute and related code, that one looks better, specifically the related msbuild targets. I'll have to try those and get back to you.

I tried out the new testing infrastructure, with WebApplicationFactory et al.
I got it working for the tutorial of the page above, but for my application it's not a good fit.

No proper support for Kestrel, HttpSys, I know there is the fake TestServer approach -> https://github.com/aspnet/Mvc/issues/7818
No async init support, seeding data with some sync over async approach isn't what I consider a valid approach (Xunit has IAsyncLifetime for that btw.)

I ran into a few additional problems where the Fixture class is actually in a shared third project, but that might be just my rather complex project setup.
In any case, you can close this issue as from my pov https://docs.microsoft.com/aspnet/core/test/integration-tests documents the current approach well enough.

@javiercn, there's testing feedback for you here.

@Tornhoof I hope you found a good solution for your scenario.
Your OP here has feedback regarding:

GetProjectPath from the Testing controllers topic Accessing views section. That approach and content has been removed in the time since you opened this issue. You mentioned that UseSolutionRelativeContentRoot (from the Integration tests topic) relies on about the same approach internally.
A fixed folder layout assumption by the engineers.
Build agent scenario.

If your request is still valid feedback for the engineers, you can re-open this issue at aspnet/AspNetCore issues. You'd need to re-cast the explanation a bit given that the content you referenced before isn't available any longer.
AFAICT, we don't have anything actionable based on guidance from engineering at this time. If they come up with something, we'll open a new issue to cover it. Thanks again for opening this issue and discussing your scenario here.


listen router change ?

'fraid not. There's a proposal though: https://github.com/hyperapp/router/issues/53

Another one following the Hyperapp V2 new API : #83


Issue to trigger new releases

@JuliaRegistrator register

@JuliaRegister register

@JuliaRegister register

@Juliaregistrator register

Registration pull request created: JuliaRegistries/General/34818
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.3 -m "<description of version>" 7923b0bb531d8fb354842b4a68d6e47c41d15e66
git push origin v0.1.3

@JuliaRegistrator register branch=master
cc @devmotion

Registration pull request created: JuliaRegistries/General/40235
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.4 -m "<description of version>" c252944fd4f438a150f3d997fefd304ed73d45cc
git push origin v0.1.4

@JuliaRegistrator register branch=master

Registration pull request created: JuliaRegistries/General/52885
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.5 -m "<description of version>" aa6031f4759fdf95ea2542215405a10e8ebee37c
git push origin v0.1.5

cc @devmotion

@JuliaRegistrator register branch=master
Release notes:

Add detailed MLJ-compliant document strings (#8)
Replace under-the-hood permutedims for incoming features, with adjoints (#10)

Registration pull request created: JuliaRegistries/General/66464
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.6 -m "<description of version>" cad984bb473b4d3a4bdccd56877482c40a253669
git push origin v0.1.6


tf.data.Iterator has_next(), next(), feature request

Request:
has_next()  // opt that check if next data is present at graph execution time without resorting to python try/except
next()  // op that advance the iterator by one at graph execution time
Reason:
has_next():
Iterator seems to be designed for supervised learning. Data is fed from one end, OutOfRangeError occurs when all the data has been used and training stops (if repeat is not set).
In reinforcement learning, the end of episode (last example in file) can be a significant source of information during graph execution time. Catch the exception and run an additional ad-hoc training cycle for the last example is aweful.
next():
In RL, the data can be sequential and has specific time stamps. During graph execution time, the environment written in tensorflow should be able to advance the Iterator using tf.while_loop with iterator.next() until some condition is met
for example, advance Iterator in a tf.while_loop until time stamp associated with that example is greater than some internal state (variable).
Perhaps the get_next() method should be parameter such that the node can be optionally required to re-executed after every visit. or just have a next() method and let us figure out the appropriate dependency control.
Have I written custom code: Yes
OS Platform and Distribution: Ubuntu 16.04 LTS
TensorFlow installed from: pip
TensorFlow version: 1.8
Bazel version: 0.15.2
CUDA/cuDNN version: Cuda 9.0, cuDNN v7
GPU model and memory: GTX 1070 8GB
Exact command to reproduce: N/A
Mobile device: N/A

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce
Mobile device

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

I recently added tf.contrib.data.get_next_as_optional() (9811b72df7347a4ec2ee884a2165775340d70e75), which makes it possible to consume values from an iterator in a tf.while_loop() without raising an exception at the end of the sequence. This should make it possible to implement your RL use case.
As I commented on #20255, we have no plans to add Iterator.has_next(), but a combination of tf.contrib.data.get_next_as_optional() and local variables should suffice to implement your use case.


Refocus has not been updated

Refocus was updated in the remaster. The requirement reads, "You have a focus pool." In the system it currently reads, "You have a focus pool, and you have spent at least 1 Focus Point since you last regained any Focus Points."

This was fixed with https://github.com/foundryvtt/pf2e/pull/11704.


[KOGITO-5104] - [1.5.x] - RHPAM Kogito operator creates wrong default image tags

Many thanks for submitting your Pull Request :heart:!
https://issues.redhat.com/browse/KOGITO-5104
Please make sure your PR meets the following requirements:

[ ] You have read the contributors' guide
[ ] Pull Request title is properly formatted: [KOGITO-XYZ] Subject
[ ] Pull Request contains a link to the JIRA issue
[ ] Pull Request contains a description of the issue
[ ] Pull Request does not include fixes for issues other than the main ticket
[ ] Your feature/bug fix has a unit test that verifies it
[ ] You've tested the new feature/bug fix in an actual OpenShift cluster
[ ] You've added a RELEASE_NOTES.md entry regarding this change

Codecov Report

:exclamation: No coverage uploaded for pull request base (1.5.x@a3743bc). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             1.5.x     #35   +/-   ##
=======================================
  Coverage         ?   0.00%
=======================================
  Files            ?       3
  Lines            ?     109
  Branches         ?       0
=======================================
  Hits             ?       0
  Misses           ?     109
  Partials         ?       0

Flag
Coverage Δ

controllers
0.00% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3743bc...5baeffd. Read the comment docs.


make error while following the procedure in apache2.md

Hey, I was trying to enable eSNI on my Apache server and am following the steps as given in https://github.com/sftcd/openssl/blob/master/esnistuff/apache2.md
After "./configure --enable-ssl --with-ssl=$HOME/code/openssl-for-apache --with-libxml2" when i try make, it gives the following error:
......warnings.......

ssl_engine_init.c:334:17: error: too few arguments to function ‘SSL_CTX_esni_server_enable’
             if (SSL_CTX_esni_server_enable(ctx,privname,pubname)!=1) {
                 ^~~~~~~~~~~~~~~~~~~~~~~~~~

......warnings.......

/home/azureuser/code/httpd/build/rules.mk:213: recipe for target 'ssl_engine_init.slo' failed
make[4]: *** [ssl_engine_init.slo] Error 1
make[4]: Leaving directory '/home/azureuser/code/httpd/modules/ssl'
/home/azureuser/code/httpd/build/rules.mk:117: recipe for target 'shared-build-recursive' failed
make[3]: *** [shared-build-recursive] Error 1
make[3]: Leaving directory '/home/azureuser/code/httpd/modules/ssl'
/home/azureuser/code/httpd/build/rules.mk:117: recipe for target 'shared-build-recursive' failed
make[2]: *** [shared-build-recursive] Error 1
make[2]: Leaving directory '/home/azureuser/code/httpd/modules'
/home/azureuser/code/httpd/build/rules.mk:117: recipe for target 'shared-build-recursive' failed
make[1]: *** [shared-build-recursive] Error 1
make[1]: Leaving directory '/home/azureuser/code/httpd'
/home/azureuser/code/httpd/build/rules.mk:75: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

There are a fair few warnings as well:
ssl_engine_init.c: In function ‘make_dh_params’:
ssl_engine_init.c:110:5: warning: ‘DH_new’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
     DH *dh = DH_new();
     ^~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:199:27: note: declared here
 OSSL_DEPRECATEDIN_3_0 DH *DH_new(void);
                           ^~~~~~
ssl_engine_init.c:121:5: warning: ‘DH_set0_pqg’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
     if (!p || !g || !DH_set0_pqg(dh, p, NULL, g)) {
     ^~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:255:27: note: declared here
 OSSL_DEPRECATEDIN_3_0 int DH_set0_pqg(DH *dh, BIGNUM *p, BIGNUM *q, BIGNUM *g);
                           ^~~~~~~~~~~
ssl_engine_init.c:122:9: warning: ‘DH_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         DH_free(dh);
         ^~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:200:28: note: declared here
 OSSL_DEPRECATEDIN_3_0 void DH_free(DH *dh);
                            ^~~~~~~
ssl_engine_init.c: In function ‘free_dh_params’:
ssl_engine_init.c:160:9: warning: ‘DH_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         DH_free(dhparams[n].dh);
         ^~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:200:28: note: declared here
 OSSL_DEPRECATEDIN_3_0 void DH_free(DH *dh);
                            ^~~~~~~
ssl_engine_init.c: In function ‘load_esnikeys’:
ssl_engine_init.c:334:48: warning: passing argument 2 of ‘SSL_CTX_esni_server_enable’ from incompatible pointer type [-Wincompatible-pointer-types]
             if (SSL_CTX_esni_server_enable(ctx,privname,pubname)!=1) {
                                                ^~~~~~~~
In file included from ssl_engine_init.c:40:0:
/home/azureuser/code/openssl-for-apache/include/openssl/esni.h:545:5: note: expected ‘SSL * {aka struct ssl_st *}’ but argument is of type ‘char *’
 int SSL_CTX_esni_server_enable(SSL_CTX *s, SSL *con, const char *esnikeyfile, const char *esnipubfile);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c:334:17: error: too few arguments to function ‘SSL_CTX_esni_server_enable’
             if (SSL_CTX_esni_server_enable(ctx,privname,pubname)!=1) {
                 ^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_engine_init.c:40:0:
/home/azureuser/code/openssl-for-apache/include/openssl/esni.h:545:5: note: declared here
 int SSL_CTX_esni_server_enable(SSL_CTX *s, SSL *con, const char *esnikeyfile, const char *esnipubfile);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_Engine’:
ssl_engine_init.c:633:9: warning: ‘ENGINE_by_id’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         if (!(e = ENGINE_by_id(mc->szCryptoDevice))) {
         ^~
In file included from ssl_private.h:105:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:336:31: note: declared here
 OSSL_DEPRECATEDIN_3_0 ENGINE *ENGINE_by_id(const char *id);
                               ^~~~~~~~~~~~
ssl_engine_init.c:643:13: warning: ‘ENGINE_ctrl’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
             ENGINE_ctrl(e, ENGINE_CTRL_CHIL_SET_FORKCHECK, 1, 0, 0);
             ^~~~~~~~~~~
In file included from ssl_private.h:105:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:429:27: note: declared here
 OSSL_DEPRECATEDIN_3_0 int ENGINE_ctrl(ENGINE *e, int cmd, long i, void *p,
                           ^~~~~~~~~~~
ssl_engine_init.c:647:9: warning: ‘ENGINE_set_default’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         if (!ENGINE_set_default(e, ENGINE_METHOD_ALL)) {
         ^~
In file included from ssl_private.h:105:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:708:27: note: declared here
 OSSL_DEPRECATEDIN_3_0 int ENGINE_set_default(ENGINE *e, unsigned int flags);
                           ^~~~~~~~~~~~~~~~~~
ssl_engine_init.c:658:9: warning: ‘ENGINE_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         ENGINE_free(e);
         ^~~~~~~~~~~
In file included from ssl_private.h:105:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:493:27: note: declared here
 OSSL_DEPRECATEDIN_3_0 int ENGINE_free(ENGINE *e);
                           ^~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_ctx_tls_extensions’:
ssl_engine_init.c:736:9: warning: ‘SRP_VBASE_new’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         if (!(mctx->srp_vbase = SRP_VBASE_new(mctx->srp_unknown_user_seed))) {
         ^~
In file included from ssl_private.h:220:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/srp.h:173:12: note: declared here
 SRP_VBASE *SRP_VBASE_new(char *seed_key);
            ^~~~~~~~~~~~~
ssl_engine_init.c:745:9: warning: ‘SRP_VBASE_init’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         err = SRP_VBASE_init(mctx->srp_vbase, mctx->srp_vfile);
         ^~~
In file included from ssl_private.h:220:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/srp.h:177:5: note: declared here
 int SRP_VBASE_init(SRP_VBASE *vb, char *verifier_file);
     ^~~~~~~~~~~~~~
ssl_engine_init.c:753:9: warning: ‘SSL_CTX_set_srp_username_callback’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         SSL_CTX_set_srp_username_callback(mctx->ssl_ctx,
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_private.h:90:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:1959:5: note: declared here
 int SSL_CTX_set_srp_username_callback(SSL_CTX *ctx,
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c:755:9: warning: ‘SSL_CTX_set_srp_cb_arg’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         SSL_CTX_set_srp_cb_arg(mctx->ssl_ctx, mctx);
         ^~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_private.h:90:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:1961:27: note: declared here
 OSSL_DEPRECATEDIN_3_0 int SSL_CTX_set_srp_cb_arg(SSL_CTX *ctx, void *arg);
                           ^~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_ctx_callbacks’:
ssl_engine_init.c:1078:5: warning: ‘SSL_CTX_set_tmp_dh_callback’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
     SSL_CTX_set_tmp_dh_callback(ctx,  ssl_callback_TmpDH);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_private.h:90:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:2286:6: note: declared here
 void SSL_CTX_set_tmp_dh_callback(SSL_CTX *ctx,
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_server_certs’:
ssl_engine_init.c:1716:9: warning: ‘DH_bits’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s, APLOGNO(02540)
         ^~~~~~~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:202:27: note: declared here
 OSSL_DEPRECATEDIN_3_0 int DH_bits(const DH *dh);
                           ^~~~~~~
ssl_engine_init.c:1719:9: warning: ‘DH_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         DH_free(dhparams);
         ^~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:200:28: note: declared here
 OSSL_DEPRECATEDIN_3_0 void DH_free(DH *dh);
                            ^~~~~~~
ssl_engine_init.c:1729:9: warning: ‘EC_KEY_new_by_curve_name’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         && (eckey = EC_KEY_new_by_curve_name(nid))) {
         ^~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:33:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ec.h:994:31: note: declared here
 OSSL_DEPRECATEDIN_3_0 EC_KEY *EC_KEY_new_by_curve_name(int nid);
                               ^~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c:1751:5: warning: ‘EC_KEY_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
     EC_KEY_free(eckey);
     ^~~~~~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:33:0,
                 from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
                 from ssl_private.h:90,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ec.h:999:28: note: declared here
 OSSL_DEPRECATEDIN_3_0 void EC_KEY_free(EC_KEY *key);
                            ^~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_ctx_cleanup’:
ssl_engine_init.c:2008:9: warning: ‘SRP_VBASE_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
         SRP_VBASE_free(mctx->srp_vbase);
         ^~~~~~~~~~~~~~
In file included from ssl_private.h:220:0,
                 from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/srp.h:175:6: note: declared here
 void SRP_VBASE_free(SRP_VBASE *vb);
      ^~~~~~~~~~~~~~

Is this a mistake on my end or the code itself? And how can I try and fix this?
Thank you

I forgot to add the versions of the software sorry
OpenSSL: OpenSSL 1.1.1  11 Sep 2018
Apache Server: Apache/2.4.29 (Ubuntu)
Ubuntu: Ubuntu 18.04.5 LTS

Hiya,
On 26/03/2021 07:51, Dhruv Rauthan wrote:

Hey, I was trying to enable eSNI on my Apache server and am following the steps as given in https://github.com/sftcd/openssl/blob/master/esnistuff/apache2.md

It's worth noting that ESNI has evolved into ECH over
the last year, and so e.g. I don't think firefox nightly
now includes support for ESNI, but will hopefully add ECH
soonish. (There is NSS code for ECH but afaik not yet
integrated with ff.)
As a bit of background: ESNI just encrypted the SNI in a
TLS handshake extension, whereas ECH (Encrypted ClientHello)
allows for encryption of an entire "inner" ClientHello within
an "outer" ClientHello, and seems overall a better (if more
complex) design.
All that being to say that if you're doing this in an
expectation that ESNI will be part of e.g. browsers in
the future, that might not be a great plan. ECH however
isn't yet present in browsers as we're just at the point
of ironing out the last interoperability issues. But
the hope is that ECH will become part of browsers and
web servers going forward.

After "./configure --enable-ssl --with-ssl=$HOME/code/openssl-for-apache --with-libxml2" when i try make, it gives the following error:
......warnings.......

ssl_engine_init.c:334:17: error: too few arguments to function ‘SSL_CTX_esni_server_enable’
              if (SSL_CTX_esni_server_enable(ctx,privname,pubname)!=1) {

If (given the above) you still want to try this out, I can
take a look at that as I'm nearly back at the point of doing
application integrations for ECH again.
All the deprecations below are basically changes to upstream
over the last year. I've not yet bothered to fix them as the
ESNI code in my fork is likely to disappear once ECH is a bit
more developed. For now, I've left it in just in case;-)
So if you're happy to wait a month(-ish) more and then try
out ECH, that'd maybe be a more useful thing for you longer
term. If you have a pressing need for trying out ESNI, I'd be
happy to look at the error above but it probably doesn't make
sense to try fix the deprecation warnings.
Cheers,
S.

              ^~~~~~~~~~~~~~~~~~~~~~~~~~

......warnings.......
/home/azureuser/code/httpd/build/rules.mk:213: recipe for target 'ssl_engine_init.slo' failed
make[4]: *** [ssl_engine_init.slo] Error 1
make[4]: Leaving directory '/home/azureuser/code/httpd/modules/ssl'
/home/azureuser/code/httpd/build/rules.mk:117: recipe for target 'shared-build-recursive' failed
make[3]: *** [shared-build-recursive] Error 1
make[3]: Leaving directory '/home/azureuser/code/httpd/modules/ssl'
/home/azureuser/code/httpd/build/rules.mk:117: recipe for target 'shared-build-recursive' failed
make[2]: *** [shared-build-recursive] Error 1
make[2]: Leaving directory '/home/azureuser/code/httpd/modules'
/home/azureuser/code/httpd/build/rules.mk:117: recipe for target 'shared-build-recursive' failed
make[1]: *** [shared-build-recursive] Error 1
make[1]: Leaving directory '/home/azureuser/code/httpd'
/home/azureuser/code/httpd/build/rules.mk:75: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

There are a fair few warnings as well:

ssl_engine_init.c: In function ‘make_dh_params’:
ssl_engine_init.c:110:5: warning: ‘DH_new’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
DH *dh = DH_new();
^~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:199:27: note: declared here
OSSL_DEPRECATEDIN_3_0 DH *DH_new(void);
^~~~~~
ssl_engine_init.c:121:5: warning: ‘DH_set0_pqg’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
if (!p || !g || !DH_set0_pqg(dh, p, NULL, g)) {
^~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:255:27: note: declared here
OSSL_DEPRECATEDIN_3_0 int DH_set0_pqg(DH *dh, BIGNUM *p, BIGNUM *q, BIGNUM *g);
^~~~~~~~~~~
ssl_engine_init.c:122:9: warning: ‘DH_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
DH_free(dh);
^~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:200:28: note: declared here
OSSL_DEPRECATEDIN_3_0 void DH_free(DH *dh);
^~~~~~~
ssl_engine_init.c: In function ‘free_dh_params’:
ssl_engine_init.c:160:9: warning: ‘DH_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
DH_free(dhparams[n].dh);
^~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:200:28: note: declared here
OSSL_DEPRECATEDIN_3_0 void DH_free(DH *dh);
^~~~~~~
ssl_engine_init.c: In function ‘load_esnikeys’:
ssl_engine_init.c:334:48: warning: passing argument 2 of ‘SSL_CTX_esni_server_enable’ from incompatible pointer type [-Wincompatible-pointer-types]
if (SSL_CTX_esni_server_enable(ctx,privname,pubname)!=1) {
^~~~~~~~
In file included from ssl_engine_init.c:40:0:
/home/azureuser/code/openssl-for-apache/include/openssl/esni.h:545:5: note: expected ‘SSL * {aka struct ssl_st *}’ but argument is of type ‘char *’
int SSL_CTX_esni_server_enable(SSL_CTX *s, SSL *con, const char *esnikeyfile, const char *esnipubfile);
^~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c:334:17: error: too few arguments to function ‘SSL_CTX_esni_server_enable’
if (SSL_CTX_esni_server_enable(ctx,privname,pubname)!=1) {
^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_engine_init.c:40:0:
/home/azureuser/code/openssl-for-apache/include/openssl/esni.h:545:5: note: declared here
int SSL_CTX_esni_server_enable(SSL_CTX *s, SSL *con, const char *esnikeyfile, const char *esnipubfile);
^~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_Engine’:
ssl_engine_init.c:633:9: warning: ‘ENGINE_by_id’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
if (!(e = ENGINE_by_id(mc->szCryptoDevice))) {
^~
In file included from ssl_private.h:105:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:336:31: note: declared here
OSSL_DEPRECATEDIN_3_0 ENGINE *ENGINE_by_id(const char *id);
^~~~~~~~~~~~
ssl_engine_init.c:643:13: warning: ‘ENGINE_ctrl’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
ENGINE_ctrl(e, ENGINE_CTRL_CHIL_SET_FORKCHECK, 1, 0, 0);
^~~~~~~~~~~
In file included from ssl_private.h:105:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:429:27: note: declared here
OSSL_DEPRECATEDIN_3_0 int ENGINE_ctrl(ENGINE *e, int cmd, long i, void *p,
^~~~~~~~~~~
ssl_engine_init.c:647:9: warning: ‘ENGINE_set_default’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
if (!ENGINE_set_default(e, ENGINE_METHOD_ALL)) {
^~
In file included from ssl_private.h:105:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:708:27: note: declared here
OSSL_DEPRECATEDIN_3_0 int ENGINE_set_default(ENGINE *e, unsigned int flags);
^~~~~~~~~~~~~~~~~~
ssl_engine_init.c:658:9: warning: ‘ENGINE_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
ENGINE_free(e);
^~~~~~~~~~~
In file included from ssl_private.h:105:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/engine.h:493:27: note: declared here
OSSL_DEPRECATEDIN_3_0 int ENGINE_free(ENGINE *e);
^~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_ctx_tls_extensions’:
ssl_engine_init.c:736:9: warning: ‘SRP_VBASE_new’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
if (!(mctx->srp_vbase = SRP_VBASE_new(mctx->srp_unknown_user_seed))) {
^~
In file included from ssl_private.h:220:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/srp.h:173:12: note: declared here
SRP_VBASE *SRP_VBASE_new(char *seed_key);
^~~~~~~~~~~~~
ssl_engine_init.c:745:9: warning: ‘SRP_VBASE_init’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
err = SRP_VBASE_init(mctx->srp_vbase, mctx->srp_vfile);
^~~
In file included from ssl_private.h:220:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/srp.h:177:5: note: declared here
int SRP_VBASE_init(SRP_VBASE *vb, char *verifier_file);
^~~~~~~~~~~~~~
ssl_engine_init.c:753:9: warning: ‘SSL_CTX_set_srp_username_callback’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
SSL_CTX_set_srp_username_callback(mctx->ssl_ctx,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_private.h:90:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:1959:5: note: declared here
int SSL_CTX_set_srp_username_callback(SSL_CTX *ctx,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c:755:9: warning: ‘SSL_CTX_set_srp_cb_arg’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
SSL_CTX_set_srp_cb_arg(mctx->ssl_ctx, mctx);
^~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_private.h:90:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:1961:27: note: declared here
OSSL_DEPRECATEDIN_3_0 int SSL_CTX_set_srp_cb_arg(SSL_CTX *ctx, void *arg);
^~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_ctx_callbacks’:
ssl_engine_init.c:1078:5: warning: ‘SSL_CTX_set_tmp_dh_callback’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
SSL_CTX_set_tmp_dh_callback(ctx,  ssl_callback_TmpDH);
^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ssl_private.h:90:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:2286:6: note: declared here
void SSL_CTX_set_tmp_dh_callback(SSL_CTX *ctx,
^~~~~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_server_certs’:
ssl_engine_init.c:1716:9: warning: ‘DH_bits’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s, APLOGNO(02540)
^~~~~~~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:202:27: note: declared here
OSSL_DEPRECATEDIN_3_0 int DH_bits(const DH *dh);
^~~~~~~
ssl_engine_init.c:1719:9: warning: ‘DH_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
DH_free(dhparams);
^~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/dsa.h:51:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:37,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/dh.h:200:28: note: declared here
OSSL_DEPRECATEDIN_3_0 void DH_free(DH *dh);
^~~~~~~
ssl_engine_init.c:1729:9: warning: ‘EC_KEY_new_by_curve_name’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
&& (eckey = EC_KEY_new_by_curve_name(nid))) {
^~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:33:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ec.h:994:31: note: declared here
OSSL_DEPRECATEDIN_3_0 EC_KEY *EC_KEY_new_by_curve_name(int nid);
^~~~~~~~~~~~~~~~~~~~~~~~
ssl_engine_init.c:1751:5: warning: ‘EC_KEY_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
EC_KEY_free(eckey);
^~~~~~~~~~~
In file included from /home/azureuser/code/openssl-for-apache/include/openssl/x509.h:33:0,
from /home/azureuser/code/openssl-for-apache/include/openssl/ssl.h:31,
from ssl_private.h:90,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/ec.h:999:28: note: declared here
OSSL_DEPRECATEDIN_3_0 void EC_KEY_free(EC_KEY *key);
^~~~~~~~~~~
ssl_engine_init.c: In function ‘ssl_init_ctx_cleanup’:
ssl_engine_init.c:2008:9: warning: ‘SRP_VBASE_free’ is deprecated: Since OpenSSL 3.0 [-Wdeprecated-declarations]
SRP_VBASE_free(mctx->srp_vbase);
^~~~~~~~~~~~~~
In file included from ssl_private.h:220:0,
from ssl_engine_init.c:29:
/home/azureuser/code/openssl-for-apache/include/openssl/srp.h:175:6: note: declared here
void SRP_VBASE_free(SRP_VBASE *vb);
^~~~~~~~~~~~~~

Is this a mistake on my end or the code itself? And how can I try and fix this?

Thank you

Hey,
I looked into ECH over the past few days and understand that it will be the standard going into the future.
The project I am working on currently involves the decryption of the ESNI field, I believe the method will carry over to ECH (just a bit more complex) since SNI along with other fields will be encrypted there.
My current goal is to work with the decrypted ESNI value, and hence I would be extremely grateful if you could look into the error whenever possible
Thank you

ESNI stuff is legacy nowadays: closing


Use multiprocessing join.

Checklist before submitting

[x] Did you read the contributor guide?

Description
Fixes #60

Apparently the join does some synchronizing that pytest-cov hooks into to do some cleanup and combining of coverage results.


[Bug] Make ESLint Work

Currently, ESlint is not catching any errors. Check and fix it.

Turns out it's due to eslint disabled on specific files.


VE Failing to retrieve license

@richbrowne --> This is the issue related to my slack message
After being patched with the patch-image tool and booted with a heat stack, the BIG-IP fails to license properly via the startup script and the openstack functions.
The license failure message is as follows from /var/log/message:
Apr  2 23:08:34 host-10-4-4-172 notice openstack-init: Saving Ethernet mapping...done Apr  2 23:08:34 host-10-4-4-172 notice openstack-init: Licensing BIG-IP using license key LYMWS-IUKBJ-GEKAP-XMBSE-LVBRHXN... Apr  2 23:08:40 host-10-4-4-172 notice openstack-init: Unknown exception Apr  2 23:08:40 host-10-4-4-172 notice openstack-init: Could not license BIG-IP (attempt #1/5), retrying in 5 seconds... Apr  2 23:08:45 host-10-4-4-172 notice openstack-init: Licensing BIG-IP using license key LYMWS-IUKBJ-GEKAP-XMBSE-LVBRHXN... Apr  2 23:08:48 host-10-4-4-172 notice openstack-init: Unknown exception Apr  2 23:08:48 host-10-4-4-172 notice openstack-init: Could not license BIG-IP (attempt #2/5), retrying in 5 seconds... Apr  2 23:08:50 host-10-4-4-172 notice setsebool: The allow_dynad policy boolean was changed to on by root Apr  2 23:08:53 host-10-4-4-172 notice openstack-init: Licensing BIG-IP using license key LYMWS-IUKBJ-GEKAP-XMBSE-LVBRHXN... Apr  2 23:08:57 host-10-4-4-172 notice openstack-init: Unknown exception Apr  2 23:08:57 host-10-4-4-172 notice openstack-init: Could not license BIG-IP (attempt #3/5), retrying in 5 seconds... Apr  2 23:09:02 host-10-4-4-172 notice openstack-init: Licensing BIG-IP using license key LYMWS-IUKBJ-GEKAP-XMBSE-LVBRHXN... Apr  2 23:09:06 host-10-4-4-172 notice openstack-init: Unknown exception Apr  2 23:09:06 host-10-4-4-172 notice openstack-init: Could not license BIG-IP (attempt #4/5), retrying in 5 seconds... Apr  2 23:09:11 host-10-4-4-172 notice openstack-init: Licensing BIG-IP using license key LYMWS-IUKBJ-GEKAP-XMBSE-LVBRHXN... Apr  2 23:09:14 host-10-4-4-172 notice syslog-ng[10024]: Configuration reload request received, reloading configuration; Apr  2 23:09:20 host-10-4-4-172 notice openstack-init: Unknown exception Apr  2 23:09:20 host-10-4-4-172 notice openstack-init: Failed to license BIG-IP after 5 attempts, quitting... Apr  2 23:09:20 host-10-4-4-172 notice openstack-init: Could not locate valid BIG-IP license file, no licensed modules found... Apr  2 23:10:03 host-10-4-4-172 notice openstack-init: Successfully connected to mcpd... Apr  2 23:10:03 host-10-4-4-172 notice openstack-init: Successfully connected to mcpd... Apr  2 23:10:03 host-10-4-4-172 notice openstack-init: detected tmm started Apr  2 23:10:04 host-10-4-4-172 notice openstack-init: Saving running configuration... Apr  2 23:10:04 host-10-4-4-172 notice openstack-init:   /config/bigip.conf Apr  2 23:10:04 host-10-4-4-172 notice openstack-init:   /config/bigip_base.conf Apr  2 23:10:04 host-10-4-4-172 notice openstack-init:   /config/bigip_user.conf Apr  2 23:10:04 host-10-4-4-172 notice openstack-init: Saving Ethernet mapping...done Apr  2 23:10:09 host-10-4-4-172 notice openstack-init: Completed OpenStack auto-configuration in 162 seconds... Apr  2 23:10:24 host-10-4-4-172 notice setsebool: The allow_dynad policy boolean was changed to on by root

This appears to have been a transient problem with the F5 license server. It has not reared its head in the past two weeks. Closing.


[Q] [v4] Is it safe to change to gutter without touching the breakpoints?

I like to change the gutter between columns, replacing 15px with my value (e.g: 10px) is enough or should I change the breakpoints too?
I asked this question because of following discussions:
https://github.com/twbs/bootstrap/issues/21333
https://github.com/twbs/bootstrap/issues/18054
https://github.com/twbs/bootstrap/issues/14894

You don't need to change the breakpoints unless you want.

@mdo I changed the gutter to 0px, now cols doesn't fill the width of the container.


Encrypt Sorted Recovery files

During recovery, the write ahead logs are read, sorted and written back to disk using org.apache.hadoop.io.MapFile.Writer(). If the On Disk Encryption feature is used, the sorted Map files will not be encrypted during this time. In order for the On Disk Encryption to be complete, these files need to be encrypted on disk. The best solution would probably be to use RFile, like is used with the rest of Accumulo.
Code dealing with sorting:
https://github.com/apache/accumulo/blob/bbd87a6693fc5cdbbe947b0821a4f05a18cf905b/server/tserver/src/main/java/org/apache/accumulo/tserver/log/LogSorter.java#L177
https://github.com/apache/accumulo/blob/7e3c67d0c0b831d7d29c69e24d07dd700f24c5e0/server/manager/src/main/java/org/apache/accumulo/manager/recovery/RecoveryManager.java#L151
Then tserver will recover the sorted logs:
https://github.com/apache/accumulo/blob/30ce59fd94f51b60a30ced328156f02d3223330b/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java#L357-L358
https://github.com/apache/accumulo/blob/6a74b4667e3bd33e34b5262c5dd8ea64167fb657/server/tserver/src/main/java/org/apache/accumulo/tserver/log/SortedLogRecovery.java#L285-L286

I can see where that would speed up recovery. I haven't done much with locality groups so not sure how that would work with the sorting of the log events. I was going with a schema that preserves the current way the WAL events sort, by Event + tabletId + sequence number:

        Row = EventTypeInteger_tabletID_seq
        Family = event
        Qualifier = tserverSession OR filename OR KeyExtent

I created the Event type integer based on what is returned by the compareTo() of LogFileKey. Then appended the tabletId integer (which is written from commitSession.getLogId()) and finally the sequence number. The event type determines what will be stored in the column qualifier.
https://github.com/apache/accumulo/blob/583ca55ceb5402d6bbd0fb093feccc897234582f/server/tserver/src/main/java/org/apache/accumulo/tserver/logger/LogFileKey.java#L155-L164
I was using scanner.fetchColumnFamily(cf); where the cf could be 0 or many Text... colFamToFetch and passing in the different events to fetch. Maybe I can put up a draft PR of just the LogFileKey.toKey() method so you can provide some feedback. There are already a good number of changes but its mostly changing all the iterators to work with Key/Value.

If I gave each enum in LogEvents a unique number for sorting, then I wouldn't have to store the string in the column family. The new numbering could look like this:

    // OPEN = 0
    // DEFINE_TABLET = 1
    // COMPACTION_START = 2
    // COMPACTION_FINISH = 3
    // MUTATION = 4
    // MANY_MUTATIONS = 5

It would be extra convenient if the ordinal() of the enum corresponded to these numbers but currently that is not the case. The sorted WALs between previous versions and this one will already be incompatible so it may not be a big deal to change the order of the LogEvents enum. But I wonder if it could lead to some crazy bugs if a user were to recover an old WAL.

I was thinking locality groups could be used to efficiently segregate information from different tablets. So, to recover a single tablet, you don't need to scan through any of the other tablets' data.

I think the sorted WALs are structured such that needed information sorts together and can be efficiently found via seek.  I have not looked at the code in a while, but I am not sure sorted WALs are ever scanned entirely.  I think they may only be seeked to specific places.  If this is true then there may not be a benefit to LGs.

But I wonder if it could lead to some crazy bugs if a user were to recover an old WAL.

Would need to maintain code to recover old WALs, or refuse to upgrade if WALs are present forcing user to run old version and resolve WALs.  Refusing to upgrade could be annoying for the user as its difficult to know if there are WALs. They could end up in a loop trying to run new version and having to fall back to old version.

I wouldn't use an integer in place of the enum. RFile is already probably fine at compressing, and there's value in being able to actually read the file contents while troubleshooting. I don't think it would save much.
As Keith says, the current key scheme may be adequate... it's certainly the simplest migration. In that case, you don't need to use the CF, CQ, or CV at all. Just Row/Value.
Where I was going with the LG conversation was that I was just thinking that we could eliminate the tablet ID in the key structure, and make it efficient to recover an individual tablet if each tablet got its own locality group. We might even be able to eliminate the tabletID mapping and just use the extent directly, since it wouldn't be duplicated in the key. I don't see any advantage to using the event type for locality groups, though. We don't need efficient reads of all events of a specific type (I don't think so, anyway).
The wrench in my thinking is that I'm not that familiar right now with the recovery process, and I'm not actually sure why event type sorts before tablet ID.

As Keith says, the current key scheme may be adequate... it's certainly the simplest migration. In that case, you don't need to use the CF, CQ, or CV at all. Just Row/Value.

The current scheme in LogFileKey is just an implementation for WritableComparable which includes serialization/deserialization and the compareTo method. Part of this task is translating LogFileKey into an Accumulo Key so we can store it in an RFile. I am also translating LogFileValue into an Accumulo Value (but this is just the mutations serialized). We need somewhere to put all the information serialized in LogFileKey into Key, otherwise it will be lost when sorted.

We don't need efficient reads of all events of a specific type (I don't think so, anyway).

There are a few places during recovery where we iterate over the logs to get different types of information. The first time we only look for DEFINE_TABLET events in the log. This is where I got the idea to use event for CF. Check it out here: https://github.com/apache/accumulo/blob/6a74b4667e3bd33e34b5262c5dd8ea64167fb657/server/tserver/src/main/java/org/apache/accumulo/tserver/log/SortedLogRecovery.java#L105

The wrench in my thinking is that I'm not that familiar right now with the recovery process, and I'm not actually sure why event type sorts before tablet ID.

Join the club! I still don't understand tabletId and seq in LogFileKey. This comment here makes me think its tracking in each WAL: https://github.com/apache/accumulo/blob/6a74b4667e3bd33e34b5262c5dd8ea64167fb657/server/tserver/src/main/java/org/apache/accumulo/tserver/log/SortedLogRecovery.java#L290-L292

Need to finish #2197 before completing this task.

This is done so can be closed. Any work done for #2197 would be a separate task.


Enhancement for the repo

What is this issue for?
Enhancement
What are your suggetions

Github actions to run test and for linting.
Add issue template and PR template.
Add docker implementation.

@99sbr I am happy to add the following its something ok with you.

@Joel-hanson  sure. it will be great 😊


 Every expression containing an atomic group triggered a parsing error indicating that ?> is an unknown extension. Is this desired behavior by regexploit or will they be supported in further versions?

Hi,
this affects me as well. I would love to see this working.

Atomic groups currently aren't supported. I should probably make that explicit in the documentation. Atomic groups are not available in python's regular expression engine but regexploit could add some support for them.
It will be difficult to adapt the heuristic to handle them accurately, but we can start with an approximation of their behaviour with some false positives. Or begin by treating them like non-atomic groups but not causing an error. Not ideal.
While I can imagine some pathological regexes, do you have any real-world examples of ReDoS involving atomic groups?

Atomic groups are not available in python's regular expression engine

They are supported from Python 3.11 onwards (scroll down to the (?>...) section), released on Oct 24, 2022.


Update to IntelliJ IDEA 14.1.5

http://blog.jetbrains.com/idea/2015/09/intellij-idea-14-1-5-update-is-out/

Thank you for the contribution. It was merged directly as 7752eb43b601fb0a46edc02d5cb0e8f5bc9df44c to keep commit history cleaner. Your contribution is still credited to you.