OfflineCacheLifetimeInDays and ContentValidUntil not retained between requests

from @Kammy6679

Although we set PolicyDescriptor::OfflineCacheLifetimeInDays(int value) when protecting the files, we always get 0 as the value when unprotecting the files.

No matter what PolicyDescriptor::ContentValidUntil is set to at protection-time, no value is returned at unprotection-time.

Protection code
if (m_bContentExpir)
     {
         m_contentExpirDateTime = m_contentExpirDateTime.toUTC();
         time_t c_time = m_contentExpirDateTime.toTime_t();
         auto endValidation = chrono::system_clock::from_time_t(c_time);
         policyDescriptor->ContentValidUntil(endValidation);
     }

Unprotection code:
bool CMSDataInteractionLinux::GetContentValidUntil(QString &qsUntil)
{
    if(NULL == m_pUserPolicy)
        return false;

    std::chrono::time_point<std::chrono::system_clock> until_time =
            m_pUserPolicy->ContentValidUntil();

    time_t until_time_t = system_clock::to_time_t(until_time);
    std::tm* until_tm = gmtime(&until_time_t);

    char cTimeTemp[25] = {0};
    std::sprintf(cTimeTemp, "%04d/%02d/%02d %02d:%02d:00",
                 until_tm->tm_year,
                 until_tm->tm_mon,
                 until_tm->tm_mday,
                 until_tm->tm_hour,
                 until_tm->tm_min);
    std::string csTime(cTimeTemp);

    qsUntil = QString::fromStdString(csTime);
    return true;
}

@munjaldoshi thoughts?

We should completely remove the option OfflineCacheLifetimeInDays. We plan to remove the ability for an app or the user to set this value.

OfflineCacheLifetimeInDays should be replaced by a boolean (OfflineAccess). The app should set whether caching is allowed or not, but it shouldn't specify or make any assumptions about the longevity of the cache.

Gurgen is right. I forgot to mention we need a boolean. Just a nit suggestion: use the name allowOfflineAccess for the boolean for consistency with other platforms.

I've removed OfflineCacheLifetimeInDays  and added allowOfflineAccess boolean to REST UsageRestrictionsResponse and PublishCustomRequest objects (with serialize/deserialize it).
API references to OfflineCacheLifetimeInDays were changed to AllowOfflineAccess
Cache control now also checks if AllowOfflineAccess to cache responses.
Here is PR: https://github.com/AzureAD/rms-sdk-for-cpp/pull/17
@munjaldoshi, does allowOfflineAccess variable work now or you are just going to implement it in future?

@AgentRX, can you also tackle #13 with this work?

@joshgav, yes

Thanks to @AgentRX's help, here's a summary of the current usability of each of the following properties. Serialization of requests and responses is all in the one jsonserializer.cpp file.
@vladimp Can you confirm this is the expected behavior from the service side?
ContentValidUntil - If set in request is not reflected in response. This SDK does not set ContentValidUntil in request.
LicenseValidUntil - If set in request is reflected in response. This SDK does set LicenseValidUntil in request.
AllowOfflineAccess - If set in request is not (yet) reflected in response. This SDK does set AllowOfflineAccess in request. This property is incubating.
OfflineCacheLifetimeInDays - If set in request is not reflected in response. This SDK does not offer this setting. This property is deprecated.

Hi Josh!
What you defined is not accurate,  (the names on the service are a little bit confusing..)
When creating a custom policy(publishing scenario) from the policy descriptor the following properties can be passed:
Request:
{
"PreferDeprecatedAlgorithms": boolean,
"AllowAuditedExtraction": boolean,
"ReferralInfo": string,
"SignedApplicationData": {
"key": "value",
…
},
"Policy": {
"Descriptors": [ {
"Name": string,
"Description": string,
"Language": string
}],
"UserRights": [{
"Users": array of string,
"Rights": array of string }...],
"UserRoles": [{
  "Users": array of string,
  "Roles": array of string }...],

"IntervalTimeInDays": int,
"LicenseValidUntil": Time,

"EncryptedApplicationData": {
  "key": "value",
  …
}

}
Response:
{
"SerializedPublishingLicense": string,
"Id": string,
"Name": string,
"Description": string,
"Referrer": string,
"Owner": string,
"Key": {
"Value": string,
"Algorithm": string,
"CipherMode": string
}
}
When consuming the serializedLicence :
Request:
{
"SerializedPublishingLicense": string
}
Response:
{
"AccessStatus": string,
"ContentId": string,
"Id": string,
"Name": string,
"Description": string,
"Referrer": string,
"Owner": string,
"ContentValidUntil": Time
"LicenseValidUntil": Time
"Key": {
"Value": string,
"Algorithm": string,
"CipherMode": string
},
"Rights":[ string array ],
"Roles":[ string array ],
"FromTemplate": Boolean,
"SignedApplicationData": {
"key": "value",
…
},
"EncryptedApplicationData": {
"key": "value",
…
},
"IssuedTo": string,
"Policy": {
"AllowAuditedExtraction": boolean,
"UserRights": [{
  "Users": array of string,
  "Rights": array of string }...],

"UserRoles": [{
  "Users": array of string,
  "Roles": array of string }...],

"IntervalTimeInDays": int,
"LicenseValidUntil": Time

}
}
·         The Policy object will be returned only to the owner otherwise will be null.
·         The ContentValidUntil and LicenseValidUntil outside the Policy object will be always visible.
And this is the confusing part, the values that you set in publishing set as following in consumption:
IntervalTimeInDays --> LicenseValidUntil
LicenseValidUntil --> ContentValidUntil (this is exposed as a property on the userPolicy directly)
·         The SDK exposing in the public API:

     contentValiduntil  --> LicenseValidUntil (mapped to this one when creating the REST call)

     offlineCacheLifetimeInDays (should be replaced with allowOfflineAccess) --> IntervalTimeInDays (mapped to this one when creating the REST call in both cases, when exposing it as a bool, 0 for false and send nothing when true)

Policy.IntervalTimeInDays
Int
Yes
Defines the intervaltime in days.
If not present then license can be cached forever.
0: Cannot cache the use license
+ve number: Use license can only be cached for specified number of days. Client has to contact the server for re-acquiring the use license.
It is also shown in this documenthttps://microsoft.sharepoint.com/teams/RMCST/Specstore/RM Client/MSIPC-thin/REST Service/RTM/IPCv3 REST Service.docx?web=1.
Hopefully now it is more clear…
Thanks,
Vladi.
From: Josh Gavant<EMAIL_ADDRESS>Sent: Tuesday, June 30, 2015 10:42 AM
To: AzureAD/rms-sdk-for-cpp
Cc: Vladimir Postel
Subject: Re: [rms-sdk-for-cpp] OfflineCacheLifetimeInDays and ContentValidUntil not retained between requests (#5)
Thanks to @AgentRXhttps://github.com/AgentRX's help, here's a summary of the current usability of each of the following properties. Serialization of requests and responses is all in the one jsonserializer.cpphttps://github.com/AzureAD/rms-sdk-for-cpp/blob/master/sdk/rms_sdk/Json/jsonserializer.cpp file.
@vladimphttps://github.com/vladimp Can you confirm this is the expected behavior from the service side?
ContentValidUntil - If set in request is not reflected in response. This SDK does not set ContentValidUntil in request.
LicenseValidUntil - If set in request is reflected in response. This SDK does set LicenseValidUntil in request.
AllowOfflineAccess - If set in request is not (yet) reflected in response. This SDK does set AllowOfflineAccess in request. This property is incubating.
OfflineCacheLifetimeInDays - If set in request is not reflected in response. This SDK does not offer this setting. This property is deprecated.
—
Reply to this email directly or view it on GitHubhttps://github.com/AzureAD/rms-sdk-for-cpp/issues/5#issuecomment-117275812.

Support IntervalTimeInDays added
DateTime conversion error fixed
AllowOfflineAccess support added
Now all is working currectly

Here are the transitions following @AgentRX's PR. I believe we may need to update this to cause the PolicyDescriptor::AllowOfflineAccess property to propagate to the JSON payload as IntervalTimeInDays rather than as AllowOfflineAccess, as per the [*].
Once done, we can treat ContentValidUntil as the legacy expiration date. If it's used in the original publication it's the authoritative date on consumption. If however AllowOfflineAccess is used in publication, then it will become the authoritative info on consumption. The pseudocode below ensures this I believe.

PolicyDescriptor
UserPolicy
PublishCustomRequest
JsonSerializer

ContentValidUntil
ftContentValidUntil
ftLicenseValidUntil
LicenseValidUntil

AllowOfflineAccess
bAllowOfflineAccess
bAllowOfflineAccess[*]
AllowOfflineAccess[*]

?
?
?
IntervalTimeInDays

JsonSerializer
UsageRestrictionsResponse
UserPolicy (private)
UserPolicy

ContentValidUntil
ftContentValidUntil
m_ftValidityTimeUntil
ContentValidUntil

LicenseValidUntil
ftLicenseValidUntil
bAllowOfflineAccess[**]
AllowOfflineAccess

[*] Should AllowOfflineAccess be converted to IntervalTimeInDays for the JSON payload? If so, it seems the value should be time::max so that it will always translate to AllowOfflineAccess(true) on consumption, even a long time later.
[**] Here's the logic in pseudocode:
m_ftValidityTimeUntil = ftContentValidUntil
if m_ftValidityTimeUntil != ftLicenseValidUntil
    if ftLicenseValidUntil > now
        bAllowOfflineAccess = true
    else
        bAllowOfflineAccess = false

Hello @joshgav,
There are two important things of custom policy:

JSON 'Policy' is only owner view section. It's haven't been set in response for other users
There is no support for AllowOfflineAccess boolean yet

Below is the updated table:
Request

PolicyDescriptor
UserPolicy
PublishCustomRequest
JsonSerializer

ContentValidUntil
ftContentValidUntil
ftLicenseValidUntil
Policy.LicenseValidUntil

AllowOfflineAccess
bAllowOfflineAccess
bAllowOfflineAccess
Policy.AllowOfflineAccess and Policy.IntervalTimeInDays

Response

JsonSerializer
UsageRestrictionsResponse
UserPolicy (private)
UserPolicy

ContentValidUntil and Policy.LicenseValidUntil
ftContentValidUntil
m_ftValidityTimeUntil
ContentValidUntil

LicenseValidUntil and Policy.IntervalTimeInDays
ftLicenseValidUntil
bAllowOfflineAccess
AllowOfflineAccess

[*] Below is the updated logic in pseudocode:
ftValidityTimeUntil = ftContentValidUntil
bAllowOfflineAccess = true;

if(Policy.IntervalTimeInDays <= 0)
   bAllowOfflineAccess = false;

if m_ftValidityTimeUntil != ftLicenseValidUntil && ftLicenseValidUntil < now
   bAllowOfflineAccess = false

Examples:
Request
PolicyDescriptor.ContentValidUntil = 2015-07-15T15:14:50;
PolicyDescriptor.AllowOfflineAccess = false;
{"Policy":{"IntervalTimeInDays":0,"LicenseValidUntil":"2015-07-15T15:14:50", allowOfflineAccess":false}}
Response
{"ContentValidUntil":"2015-07-15T15:14:00","LicenseValidUntil":"2015-07-13T12:18:50Z",Policy":{"IntervalTimeInDays":0,"LicenseValidUntil":"2015-07-15T15:14:00"}}
Request
PolicyDescriptor.ContentValidUntil = 2015-07-15T15:28:13;
PolicyDescriptor.AllowOfflineAccess = true;
{"Policy":{"IntervalTimeInDays":1,"LicenseValidUntil":"2015-07-15T15:28:13","allowOfflineAccess":true}}
Response
{"ContentValidUntil":"2015-07-15T15:28:00","LicenseValidUntil":"2015-07-14T12:30:13Z", "Policy":{"IntervalTimeInDays":1,"LicenseValidUntil":"2015-07-15T15:28:00"}}

I've created new merged PR https://github.com/AzureAD/rms-sdk-for-cpp/pull/45

@gurgenh please review

Thanks @AgentRX. In the last Response above, should it be 2015-07-16? I.e.:
{"ContentValidUntil":"2015-07-15T15:28:00","LicenseValidUntil":"2015-07-16T12:30:13Z", "Policy":{"IntervalTimeInDays":1,"LicenseValidUntil":"2015-07-15T15:28:00"}}

What would happen if this response was received on 7/25? Would it be marked as AllowOfflineAccess=false? If so, should we set IntervalTimeInDays to int::max so that LicenseValidUntil is always > now?
I believe that AllowOfflineAccess will never be a property in the JSON payload, only in the client library. I've asked @gurgenh to clarify.

@joshgav, if I set intervalTimeInDays to int::max Azure Server returns the error:

{"Code":"Microsoft.RightsManagement.Exceptions.UnexpectedServerException","Message":"Oops. Something went wrong."}

Shall I set it to 25 years (about 9125 days)?
Also there is another problem Azure Server does:
IF Now + Request.intervalTimeInDays > Request.ContentValidUntil THEN Response.intervalTimeInDays = Request.ContentValidUntil - now
Example
PolicyDescriptor.ContentValidUntil = Now + 100 days;
PolicyDescriptor.AllowOfflineAccess = true;
Request:

{"Policy":"IntervalTimeInDays":9125,"LicenseValidUntil":"2015-10-24T18:49:17", "allowOfflineAccess":true}}

Response:

{"ContentValidUntil":"2015-10-24T18:49:00","LicenseValidUntil":"2015-10-24T18:49:00","Policy":{"IntervalTimeInDays":101,"LicenseValidUntil":"2015-10-24T18:49:00"}}

I've created new PR https://github.com/AzureAD/rms-sdk-for-cpp/pull/48

As we spoke with @vladimp IntervalTimeInDays is set to 30 days when AllowOfflineAccess
New PR: https://github.com/AzureAD/rms-sdk-for-cpp/pull/50

Please remove the AllowOfflineAccess from the UserPolicy public surface.
I confirmed that the value for intervalTimeInDays should be set to 30 when AllowOfflineAccess set to True and 0 when set to False.

Hello @vladimp,
bool AllowOfflineAccess();

Is read-only property of UserPolicy. It refers to private ProtectionPolicy object and not allowed to be changed.

Yes, We currently not exposing it in other SDK's: iOS, Mac and Android.

Using RMS SDK for Mac, we can set PolicyDescriptor::OfflineCacheLifetimeInDays(int value) when protecting the files. We cannot get the value when unprotecting the files in Mac OS. This is a bug in RMS SDK for Mac. But we can get the value when unprotecting the files in Windows. This means that the setting is right in RMS SDK for Mac.
For unity and compatibility in each RMS SDK, please  add this feature.

Thanks Kammy - as we discussed in this thread, OfflineCacheLifetimeInDays is a deprecated property and will eventually be removed from other SDKs as well. Please use ContentValidUntil instead.
It would be great if you could describe scenarios you'd like to address and the properties and behaviors you'd like to provide. The service team (@vladimp) can take that into consideration for further planning.
Thanks!

OfflineCacheLifetimeInDays and ContentValidUntil are the different methods. Please refer to MSICP in the following page and screenshot: https://msdn.microsoft.com/en-us/library/windows/desktop/hh535287(v=vs.85).aspx

Please don't remove this method from any SDK. we have some customers who use this feature in Office/Windows.

we find that:

IntervalTimeInDays is set to 30 days by default when AllowOfflineAccess. We cannot chanage this setting.
We set the ContentValidUntil, and the time is 8 hours ahead.
We can get the value of ContentValidUntil , but the value is wrong.

Hi @Kammy6679,

IntervalTimeInDays is by design set to 30 days when AllowOfflineAccess is set to true.
Please share debug logs following the instructions in the README showing the incorrect time.
Where are you getting this value and what is it?

/cc @munjaldoshi @gurgenh @vladimp @prvijay - Cache and content expiration times and how they are handled may need more synchronization across the API and all SDKs. The discrepancies make these hard to rely on currently.

IntervalTimeInDays is an internal attribute that controls the cache time to support offline scenario.
On the public surface we expose only the AllowOfflineAccess flag, we make the decision for the developer for how long we cache as this logic is not something the developer should be aware of. This value also depends on server configurations and other features.(such as revocation)
Let me know if you have any other issues or concerns.

We cannot provide log. Following are the steps to reproduce the issue:
a. The content expiration time we set is local system time: 2015-7-24 00:00:00
b. Change this time to UTC: 2015-7-23 16:00:00
c. Set ContentValidUntil as the time we get in step b.
d. In Windows, the content expiration time got by MSICP is 2015-7-24 8:00:00, which is not correct. The right value is 2015-7-24 00:00:00.

We use the follow code to get this value. If set the value is 2015-7-24 00:00:00, get the value is 0115-6-23 16:00:00(Is not a fixed value, it may change at any time)

bool CMSDataInteractionLinux::GetContentValidUntil(QString &qsUntil)
{
if(NULL == m_pUserPolicy)
return false;
std::chrono::time_point<std::chrono::system_clock> until_time =
        m_pUserPolicy->ContentValidUntil();

time_t until_time_t = system_clock::to_time_t(until_time);
std::tm* until_tm = gmtime(&until_time_t);

char cTimeTemp[25] = {0};
std::sprintf(cTimeTemp, "%04d/%02d/%02d %02d:%02d:00",
             until_tm->tm_year,
             until_tm->tm_mon,
             until_tm->tm_mday,
             until_tm->tm_hour,
             until_tm->tm_min);
std::string csTime(cTimeTemp);

qsUntil = QString::fromStdString(csTime);
return true;

}

any update?

Kemmy, as mentioned several times in the thread abov,e we want to deprecate the ability for the apps to set cache lifetime in days across all SDKs. Yes, an admin can set it on the server side but that is all we are going to allow going forward. We are going to deprecate that across all platforms in the upcoming months.
The main reason for doing tis that we do not think there is any good reason an app or a user has to worry about how long licenses live in the cache. We want to be able to change our caching characteristics without impacting the applications and the users.
So I suggest you remove the UI option from the current Foxit application to set the exact lifetime in days. You can replace it with a checkbox if you like (we do that in our own applications).

@munjaldoshi , OK, we will remove the setting of license expiration. Do you have any update about the above #2, #3

@Kammy6679, I've fixed ContentValidUntil wrong value on library side

@vladimp, There is some bug on JSON RESP API:
Request (publishinglicenses):
{"Policy":{"IntervalTimeInDays":0,"LicenseValidUntil":"2015-08-12T12:06:29Z"}}
BUT Azure Server Response (enduserlicenses):
{"ContentValidUntil":"2015-08-12T12:06:00","LicenseValidUntil":"2015-08-10T12:08:11Z", Policy":{"LicenseValidUntil":"2015-08-12T12:06:00"}}
So the response errors:

seconds are ignored
Response format in Local time (without 'Z' at end) but MUST be in UTC

Thank you. This issue has been fixed.


Create useful-links.md

Reformating needed: Markdown syntax problems, refer to this link
and add contribution guidelines


download fse-v1.1.1-node-v57-darwin-x64.tar.gz error

npm install 'git+https://github.com/xdissent/karma-chai.git' --save-dev
npm WARN gentlyRm not removing /Users/leo/RDE/laima-charon/node_modules/node-pre-gyp/node_modules/.bin/rimraf as it wasn't installed by /Users/leo/RDE/laima-charon/node_modules/node-pre-gyp/node_modules/rimraf
npm WARN gentlyRm not removing /Users/leo/RDE/laima-charon/node_modules/log4js/node_modules/.bin/semver as it wasn't installed by /Users/leo/RDE/laima-charon/node_modules/log4js/node_modules/semver
npm WARN gentlyRm not removing /Users/leo/RDE/laima-charon/node_modules/.bin/jest as it wasn't installed by /Users/leo/RDE/laima-charon/node_modules/jest-cli
npm WARN gentlyRm not removing /Users/leo/RDE/laima-charon/node_modules/cosmiconfig/node_modules/.bin/js-yaml as it wasn't installed by /Users/leo/RDE/laima-charon/node_modules/cosmiconfig/node_modules/js-yaml

<EMAIL_ADDRESS>install /Users/leo/RDE/laima-charon/node_modules/fsevents
node install

node-pre-gyp info it worked if it ends with ok
node-pre-gyp verb cli [ '/Users/leo/.nvm/versions/node/v8.0.0/bin/node',
node-pre-gyp verb cli   '/Users/leo/RDE/laima-charon/node_modules/fsevents/node_modules/.bin/node-pre-gyp',
node-pre-gyp verb cli   'install',
node-pre-gyp verb cli   '--fallback-to-build' ]
node-pre-gyp info using<EMAIL_ADDRESS>node-pre-gyp info using<EMAIL_ADDRESS>| darwin | x64
node-pre-gyp verb command install []
node-pre-gyp info check checked for "/Users/leo/RDE/laima-charon/node_modules/fsevents/lib/binding/Release/node-v57-darwin-x64/fse.node" (not found)
node-pre-gyp http GET https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz
node-pre-gyp http 404 https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz
node-pre-gyp ERR! Tried to download(404): https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for<EMAIL_ADDRESS>and<EMAIL_ADDRESS>(node-v57 ABI) (falling back to source compile with node-gyp)
node-pre-gyp http 404 status code downloading tarball https://fsevents-binaries.s3-us-west-2.amazonaws.com/v1.1.1/fse-v1.1.1-node-v57-darwin-x64.tar.gz

node 8.0.0 error, just change version to 6.x.x,


Allow usage of POST/GET variables with PHP

Severity level (1-10):
10
Files/Directories Involved:
pkg/public/servehttp.go
Description:
Something with the php-cgi command has to be changed. There is really little to no documentation that I know of over this command, but I found two stackoverflow posts that may help. I think the second of one the two is the best option.
https://stackoverflow.com/questions/4030147/how-to-pass-post-data-to-the-php-cgi
https://serverfault.com/questions/187025/how-to-pass-get-variables-to-a-php-script-via-the-command-line/187038
Personal Comments:

You may want to post here for this question => http://php.net/manual/en/install.unix.commandline.php

@KipOmaha I wanted to do that, but it says in their guidelines not to post questions in the user contributed notes, that they'll just get flagged and deleted :/

I'm sorry I wasn't being helpful, I believe it has to do with the build config you are on for php/cgi. In anycase it sounds like you need to look into ensuring you have the right build of php for the task.  :(
Also I wasn't sure which file you were talking about in your issue pkgs/public/httpserve.php wasn't there, instead there was pkgs/public/httpserve.go, I'm not to familiar with the go language.

@KipOmaha Yes you're right it should have been pkg/public/httpserve.go
I'm using the newest version of the CGI, I just think that there has to be some sort of a shell script that is written on the fly that contains certain variables like content length and the amount of post variables/etc. like its written in https://stackoverflow.com/questions/4030147/how-to-pass-post-data-to-the-php-cgi

I looked at that and found that the guy was having issues with CONTENT_LENGTH not being set.
Sorry I can't be more help, I'll probably be having this issue in the near future with cgi/python/php...so I thought I would help out and look into it.
I did find this go lib for php data bindings => https://github.com/deuill/go-php
and this issue => https://github.com/deuill/go-php/issues/51
I'm not a go guy but am a php guy, do you think go is the way or is its use specific for this project?  Maybe I should look at Go to and expirement.

Unless you're dead-set on CGI, you may consider looking into FastCGI, it's what Apache, Nginx, and Caddy now use for PHP with PHP-FPM.  It's much faster than old CGI.

@ctd1500 I was wanting to use fastCGI but when I was looking around for documentation concerning it I couldn't find anything other than webpages telling me that it was deprecated/not maintained for a long time. Do you know where I can find some documentation concerning it and its use? I would love to use it.
@KipOmaha I saw those but it looked like it wasn't full-featured when I looked at it, and it looks to still be that way. On top of that, it would always lag behind the newest version of PHP, which would be kind of unfortunate at times, so I think using some sort of CGI would probably be the way to go.

@ctd1500 Okay thank you for this information, I'll look into the libraries when I get the time and play around with it and see how it can work itself into gPanel in some way


Add page template as new page if current page is non-empty

Feature Description
As a general change of functionality of how a page template is added:

Remove the warning dialog displayed when adding a template to a non-empty page
Instead, add a new page with the page template content and switch to that page as the current page.

As a special case: If the current page matches the page template exactly, do the above as normal but also inform the user, that a new page has been added in a snackbar message. This is to inform users that something is happening every time they click to apply a page template in case they want multiple pages added with the same template.

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Implementation Brief

@barklund Hm, where do the points in the description come from? Maybe something changed since you wrote it? Or is there another link I'm missing?
Currently, in Figma, it says that it should always create a new Page no matter if previously the page is empty or not, and per examples it's always showing a snackbar message, too:

I'm following how it is in Figma, if it got updated meanwhile then that's probably correct.
Another note -- In Figma is says to set focus on two things at the same time. Since that's not possible then I'm assuming the idea is to keep the focus on the used template but select the newly added Page as the current page.

Verified in QA


Automatically load Google auth

What is the purpose of this change? What does it change?
This change removes the hardcoded Google auth mechanism for the GCS
backend, instead using Google's provided client library to discover and
generate credential material.
Google recommend that client libraries use their common auth mechanism
in order to authorise requests against Google services. Doing so means
you automatically support various types of authentication, from the
standard GOOGLE_APPLICATION_CREDENTIALS environment variable to making
use of Google's metadata API if running within Google Container Engine.
Was the change discussed in an issue or in the forum before?
Not that I'm aware of, via cursory search of issues.
Checklist

[x] I have read the Contribution Guidelines
[x] I have added tests for all changes in this PR

I don't think this is required, as we already have tests that exercise GOOGLE_APPLICATION_CREDENTIALS which will cover this and the rest of the behaviour I think we can rely on Google to have implemented properly

[x] I have added documentation for the changes (in the manual)
[x] There's a new file in a subdir of changelog/x.y.z that describe the changes for our users (template here)
[x] I have run gofmt on the code in all commits
[x] All commit messages are formatted in the same style as the other commits in the repo
[x] I'm done, this Pull Request is ready for review

@fd0 are you good on this approach? If you are then I'll fixup the conflicts- keen to get it in so more things don't change under me!

Hey, are you still interested in this PR? There's just a few minor things to edit, then I can merge it...

Hey @fd0, really sorry for the delay- this dropped off my radar since I first pr'd the change!
Will tidy this up and get it ok to merge today.

Should be good now!


AVRO schema unions code generation issue

Describe the bug
When an AVRO schema is use that contains unions, for example, ["null", "string"], the generated code is invalid.
How to Reproduce
The following snippet from an AVRO schema contains a union of "null" or "string".  See the attached bad events.yml file for the full schema.
- name: "jobOrderDescription"           doc: "JobOrderDescription"           type:           - "null"           - "string"
Run through code generator.
ag -o badevents badevents.yml @asyncapi/java-spring-cloud-stream-template
The generated POJO for the referenced event class contains the following:
public JobOrder ( 		String jobOrderId,  		<anonymous-schema-2> jobOrderDescription,  		<anonymous-schema-5> jobOrderLongDescription,  		String jobOrderNumber,  		Boolean isActive,  		String status,  		Integer statuscode)
Expected behavior
I was expecting at minimal an Object type that would allow the code to be compiled
badevents.txt
.

It seems this is a problem with the java-spring-cloud-stream-template and not the generator itself. I'd recommend closing this issue and tracking the copy of it made here https://github.com/asyncapi/java-spring-cloud-stream-template/issues/247


Only seeing 2 x TV tuners in plex after  setting up Cabernet docker, and installing Daddylive addon.

Only seeing 2 x TV tuners in plex after  setting up Cabernet docker, and installing Daddylive addon.
Is there a limit on how many tuners can use and add to Plex dvr, currently have 2 that comes come via Daddylive, yet default TV tuner count in Cabernet app is set to 10?
I am running cabernet <IP_ADDRESS>-RC01 and daddylive addon v0.1.31 which is latest one.
Any idea how this works and to add more in plex?

Number of tuners for Plex is provided by Cabernet.  Take the URL you provided to plex and add "/discover.json"  This provides the tuner count to plex.  I ran a test right now and plex is definitely providing more than 2 tuners.  You should look through the setup guide #78 it may provide info you need to solve your plex issue.

Awesome stuff, thank you.
Will check this out 😁

What's link to that setup guide for #78?


Add an AsyncRead+AsyncWrite adapter for IOStream

In many places a combination of both is required so let's make this a
bit simpler for users.

That's basically most of the boilerplate in the gio_async_tls.rs example. I'll update that once this one here is merged.

@sdroege Thanks, 👍

Thanks!


exclude roboto font

carbon is a very cool lib materialized my app with almost no extra work. but it has a side-effect that roboto fonts which i don't use in my language inceased the size of apk by 2mb. I was wondering if there's a way to remove roboto fonts from carbon lib?

I have to think about it. Basically Carbon was intended to work on Android 2.2 which is main target of my other app. Android 2.2 doesn't have Roboto, so I need it. But, I'm aware of that Roboto is present on newer systems. Maybe providing Roboto as an opt-in package would be good. The other option is to remove these files with Proguard.

I tried with proguard setting minifyEnabled to true but it doesn't seems to help

This issue was raised before with no real solution except cloning the repo and removing fonts manually. See: https://github.com/ZieIony/Carbon/issues/154

Let's keep this issue open until we find a better solution.

What if just completely remove the fonts from library package? Roboto font is free for downloading from Material Design guidelines site so anyone can get particular typefaces from the whole font archive and use them. Am I wrong?

You're right, but:

Fonts have to be in assets/fonts and have correct names. I'm sure that someone will forget about that and report not working fonts as a bug.
I'm not sure what happens when you just delete fonts. Probably it won't work on newer Androids because there's no code for using system fonts if present.

I thought that using custom fonts from assets folder is a common task and Carbon just makes it easier. I mean using fonts from assets folder of app itself and not library's.
Custom fonts can be used on any Android OS, so it seems to be a good solution for me.

Carbon contains these fonts, because carbon_Theme makes heavy use of them. I just wanted to have Roboto on all platforms and all views by default. That's why I need a specific set of correct files.
http://stackoverflow.com/questions/12128331/how-to-change-fontfamily-of-textview-in-android
Probably the best approach would be to extend android:fontFamily and android:textStyle to be translated to Roboto on all platforms (Roboto Thin working pre 4.2, etc.). The assets folder with all fonts can be moved to a different place, so users can download it right from Carbon's repository when needed. The carbon_fontPath attribute can stay as it is now.

Done. Now to get all fonts on all platforms you have to download .ttf files to your project. See samples.


Git LFS quota reached

Unfortunately, I cannot clone the repository any more because the LFS is over the data quota:
> git clone https://github.com/aquasecurity/btfhub.git
Cloning into 'btfhub'...
remote: Enumerating objects: 2586, done.
remote: Counting objects: 100% (2580/2580), done.
remote: Compressing objects: 100% (2488/2488), done.
remote: Total 2586 (delta 58), reused 2554 (delta 32), pack-reused 6
Receiving objects: 100% (2586/2586), 1.72 MiB | 3.57 MiB/s, done.
Resolving deltas: 100% (58/58), done.
Downloading archive/centos/7/arm64/3.18.9-200.el7.aarch64.btf.tar.xz (636 KB)
Error downloading object: archive/centos/7/arm64/3.18.9-200.el7.aarch64.btf.tar.xz (85841b9): Smudge error: Error downloading archive/centos/7/arm64/3.18.9-200.el7.aarch64.btf.tar.xz (85841b9e1e654a4b3a689a808c763e07085c205073b004e27946994dc449588a): batch response: This repository is over its data quota. Account responsible for LFS bandwidth should purchase more data packs to restore access.

Errors logged to /…/btfhub/.git/lfs/logs/20211105T110512.338475748.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: archive/centos/7/arm64/3.18.9-200.el7.aarch64.btf.tar.xz: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

Yes, I'm working on it, sorry about that. I'm also facing the same thing, and asked the admin to help us with that.

@saschagrunert I moved back from git-lfs to regular git (as the files wont be changing, thus the history wont have a big delta in size). I think you will be ok using the repo again, closing this. Let me know if not, please.
Cheers.


siad fails to form any contracts - unable to upload files

Bug Report
Check to see if a bug report already exists.  If you still want to create a new issue please include the information below.
Stack Trace or error message
[dymurray]$ siac renter contracts                                                     │
No contracts have been formed.

Error from minio when uploading a file:
ERRO[7005] Unexpected error, please report this issue at https://github.com/minio/minio/issues  cause=upload failed: not enough contracts to upload file: got 0, needed 20 source=[web-handlers.go:1098:toWebAPIError()]

Environment

Sia version: 1.3.1
OS: Fedora 25

Any useful logs
I'm mainly following this guide except I also am using minio as a front end for uploading files. I have also just done straight curl requests. I have enough funds I'm not certain if this behavior is expected when not enough funds are allocated to the renter. Any information you could give me would be greatly appreciated. Thanks in advance!

Can you post the contents of you contractor.log?

@ChrisSchinnerl
$ cat renter/contractor.log
2017/12/14 19:45:12.162363 contractor.go:194: STARTUP: Logging has started. Siad Version 1.3.1
2017/12/14 19:49:49.763902 contractor.go:230: SHUTDOWN: Logging has terminated.
2017/12/14 19:50:21.279342 contractor.go:194: STARTUP: Logging has started. Siad Version 1.3.1
2017/12/14 19:50:52.524975 contractor.go:230: SHUTDOWN: Logging has terminated.
2017/12/14 20:00:57.175916 contractor.go:194: STARTUP: Logging has started. Siad Version 1.3.1
2017/12/14 21:46:59.793858 contractor.go:230: SHUTDOWN: Logging has terminated.
2017/12/14 21:48:09.236364 contractor.go:194: STARTUP: Logging has started. Siad Version 1.3.1
2017/12/14 21:48:13.873272 contractor.go:230: SHUTDOWN: Logging has terminated.
2017/12/14 21:49:15.394037 contractor.go:194: STARTUP: Logging has started. Siad Version 1.3.1
2017/12/15 15:36:04.159226 allowance.go:60: INFO: setting allowance to {10 50 289 288}

@ChrisSchinnerl I build siad from source yesterday 12/14.

Your allowance seems to be extremely low (10 hastings). My guess is you wanted to set 10 SC and not 10 hastings in which case you need to multiple it by the siacoin precision as described in the "Uploading and Downloading files" section of the guide.
Keep in mind that shorter contract durations / renew windows might also make it harder to find hosts.

@ChrisSchinnerl Okay thanks for the explanation. I did in fact have the allowance lower than I wanted. I updated the renter endpoint and now have this in the log:
2017/12/15 18:04:47.108533 allowance.go:60: INFO: setting allowance to {1000000000000000000000000000 50 4320 2160}

I've let it sit for over an hour now and still haven't seen any contracts. Is this expected?

Ok this looks good now. Just let it sit for some time. It might be a bit slow atm due to the increased time between blocks. How many hosts does siac hostdb show you?

siac hostdb actually returns: No known active hosts. I'm assuming this must be my fault somehow. Really appreciate your help and quick responses!

Hosts announce themselves on the blockchain. Is your node synced?

@ChrisSchinnerl Yeah my node is synced.
Here's the GET request for consensus:
[dymurray@dymurray sia]$ curl -i -A "Sia-Agent" localhost:9980/consensus
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Fri, 15 Dec 2017 18:23:36 GMT
Content-Length: 263

{"synced":true,"height":137106,"currentblock":"0000000000000007a74458cfb8b91d7d7fab590f14d16d0072257f7e22ab3cdd","target":[0,0,0,0,0,0,0,43,165,76,234,216,132,41,88,155,108,149,204,25,249,179,120,67,19,169,194,45,225,71,76,168],"difficulty":"422647408473939000"}

Hm... is there anything suspicious in your hostdb.log ?

@ChrisSchinnerl
2017/12/14 21:49:15.386925 hostdb.go:91: STARTUP: Logging has started. Siad Version 1.3.1
2017/12/14 21:49:15.394004 scan.go:339: Performing scan on 0 online hosts and 0 offline hosts.
2017/12/14 23:16:21.158451 scan.go:339: Performing scan on 0 online hosts and 0 offline hosts.
2017/12/15 06:32:27.994825 scan.go:339: Performing scan on 0 online hosts and 200 offline hosts.
2017/12/15 09:16:00.431505 scan.go:339: Performing scan on 0 online hosts and 200 offline hosts.
2017/12/15 14:13:18.854525 scan.go:339: Performing scan on 0 online hosts and 200 offline hosts.

Interestingly a lot of the endpoints related to renter seem stale:
[dymurray@dymurray siad-docker]$ siac renter prices
Renter Prices (estimated):
  Fees for Creating a Set of Contracts:   0 H
  Download 1 TB:                          0 H
  Store 1 TB for 1 Month:                 0 H
  Upload 1 TB:                            0 H

I'm assuming that my hostdb list is either stale or pointing to invalid hosts. Just FYI I did have to manually add gateway addresses to get 1.3.1 working due to a known bug I was told. I'm not sure if that is significant but thought it was worth mentioning.

Yeah that bug should be resolved in 1.3.1RC3. Scanning for hosts might take some time. Since you don't have active hosts you could try to force a rescan by stopping siad, deleting/renaming hostdb.json and restarting siad. Afterwards let it sit for a few hours to give it time to rescan all the hosts.

Thanks @ChrisSchinnerl. I backed up hostdb.json and deleted it and restarted siad.  It took awhile to load renter and the file size is different so I know it created a new host list. I still see:
2017/12/15 19:20:25.333191 scan.go:339: Performing scan on 0 online hosts and 200 offline hosts.

But I am going to let this sit for a couple hours and will update the issue with my results.
Thanks for the help!

Yes, unfortunately it currently takes Sia several hours to scan hosts and find good ones. If you have your renter running alongside the initial blockchain sync, you typically won't have any problems because the blockchain sync also takes several hours, and the scanning happens concurrently.
But if you already have a synced consensus database, then you still have to wait several hours for the host scanning to take effect.
This is something that's on our mid-short list to fix. You should be all good if you let it run overnight once. Once you have found a few hundred good hosts, you will be all set.

@DavidVorick Thanks for the explanation, that makes sense. I'm going to let it continue running since I just started this node up yesterday and have been resetting it. I'll update the issue in a day or two if my issue is resolved.
Thank you guys for the prompt feedback!

Just wanted to update this issue. I now have 115 active hosts so that problem is solved. I still have not formed any contracts yet and the node has been up for about 3 days now. Would you guys mind telling me if my allowance is just too low?
[dymurray@dymurray app]$ siac renter allowance
Allowance:
        Amount: 1 KS
        Period: 4320 blocks

I'm sill seeing No contracts have been formed. Thanks in advance.


User shouldn't be allowed to save a resource with filetypes that aren't allowed

Already fixed because the save button is disabled until something has finished uploading and uploads aren't possible with incorrect filetypes.


Ensure transfer streams get closed on completion

A second transferOS may not be possible if the first transfer's streams do not get closed.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 70.423%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1050

Relevant Lines:
1491

💛  - Coveralls


resizing and ProPhoto RGB

First, I know very little about image theory... ;-)
We have been using imageprocessor to resize for a while now. No issues.
However, we just had an image come through that, when resized, its "color" would change (naive term, I know, but it basically got brighter):
Original Image:
Processed with: /remote.axd/path/18_raw.jpg

Resized Image:
Processed with: /remote.axd/path/18_raw.jpg**?height=700**

I started googling around and looking into exif. I found that the original image was created as a "ProPhoto RGB". I found a few articles, notably this one. After reading the general gist of "color mgmt", and this particularly:

But there are no printers and monitors which can output ProPhoto RGB. Thus, before using the image in the target medium it’s recommended to convert to the target color-space and let go off the bloat.

I thought it might have to do with what I am seeing.
Can anyone suggest if this "lightening" is expected. And suggest what should be done (ask photographer to not use ProPhoto?)
Thanks!

Hi @joshball really sorry bout the slow reply. I actually posted one a few days but it must not have actually registered on the server.
Having a look at the colourspace I would definitely expect some difference in colours. However an increased brightness is a surprise. I would have expected a decrease in tonality. Unfortunately I have no way to know what System.Drawing does when decoding an image with varying spaces.
My advice would be not use something like photoshop to convert ProPhoto to sRGB for furture web processing.
I hope that helps.
James


Rust: simplified handlers by wrapping them in a higher-order function (closure)

Refactored the solution further so now the handlers only got the bare minimum of logic. in_sell decrement and quality subtraction or addition are completely automated and part of the app infrastructure.
This leads to substantial simplified handlers, at the cost of a slightly more complicated infrastructure. The implementation if more canonical Rust code but can lead to intense fights with the borrow checker.

Nice reafctor!
" intense fights" is the main purpose of the Coding Dojo.  👍


Dataset format

💡 Your Question
I am using COCO-like dataset https://github.com/DS4SD/DocLayNet. I use COCOFormatDetectionDataset from super_gradients to load the data. According to documentation and my observation COCOFormatDetectionDataset outputs annotations in format (x, y, x, y, class_id).  I tried to use COCOFormatDetectionDataset, but training throws error:
Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
Class values must be smaller than num_classes.

and I can see that pixel bbox values are use as gt_classes values.
Tutorial in notebooks/detection_how_to_connect_custom_dataset.ipynb suggests that Trainer needs annos in format (class_id, x_center, y_center, H, W). Is it actually true, that Trainer and Datasets are not compatibile?
Should Trainer get [x_center, y_center, H, W] as un pixels values or fractions related to image size in range (0,1)?
Versions
No response

@mariannaparzych Trainer does not "expect" a specific target forma. Other component like the loss and the model do.
It would be great if you could give additional information on what it is you are trying to achieve.

@shaydeci Thank you for the aswer. You're wright, I expressed my problem incorrectly.
I am trying to use trasfer learning for odject detection on COCO-like dataset. I started with notebooks/detection_how_to_connect_custom_dataset.ipynb:
from super_gradients.training import Trainer, models, training_hyperparams

trainer = Trainer(
        experiment_name=f"{experiment_name}_{timestmp:%Y-%m-%d_%H:%M}",
        ckpt_root_dir=ckpt_root_dir,
    )
 model = models.get(
        "yolox_l", pretrained_weights="coco", num_classes=train_dataset.num_classes
    )
train_params = training_hyperparams.get("coco2017_yolox")

I am just using default configuration from the tutorial. I assumed that to load COCO-like dataset I can use COCOFormatDetectionDataset class. But it throwed error:
Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
Class values must be smaller than num_classes.
  File "/home/marianna.parzych/Unstructured/super-gradients/venv/lib/python3.10/site-packages/super_gradients/training/losses/yolox_loss.py", line 235, in _compute_loss
    gt_matched_classes, fg_mask, pred_ious_this_matching, matched_gt_inds, num_fg_img = self.get_assignments(
  File "/home/marianna.parzych/Unstructured/super-gradients/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/marianna.parzych/Unstructured/super-gradients/venv/lib/python3.10/site-packages/super_gradients/training/losses/yolox_loss.py", line 471, in get_assignments
    gt_cls_per_image = F.one_hot(gt_classes.to(torch.int64), self.num_classes)

It happens because the format of target from the dataset is wrong. I assumed, that the library has some standard formats, so you can choose dataset class that suits you and try it with different models and training parameters. If not, where can I find info in documentation about formats needed by different model implementations and loss functions?

With the COCOFormatDetectionDataset I also had issues, and I also originally expected it to just work. I think this is a bug, but maybe it's a design decision(?). It seems to load the data in XYXY format.
As a workaround, converting my data helped.
Storing the data in YOLO format and loading it with super_gradients.training.dataloaders.dataloaders.coco_detection_yolo_format_train seems to have worked for me.

I've also given the direct way a try, loading the COCO json data. This requires a bit more diving into the code:
Alternatively, you can wrap your dataset to bring it into the right shape. You'll have to determine what exactly you'll need to do for your data, of course. Something like this:

class CocoDatasetWrapper(COCOFormatDetectionDataset):
    """
	Description from Base class at https://github.com/Deci-AI/super-gradients/blob/2e591fdde09e18be06a41eafcfe7d2e8362346e4/src/super_gradients/training/datasets/detection_datasets/coco_format_detection.py#L20 :
 	Base dataset to load ANY dataset that is with a similar structure to the COCO dataset.
    - Annotation file (.json). It has to respect the exact same format as COCO, for both the json schema and the bbox format (xywh).
    - One folder with all the images.

    Output format: (x, y, x, y, class_id)
    """

    def __init__(self, fix_order=True,
                 convert_absolute_to_relative=False,
                 image_width: int|None = None,
                 image_height: int|None = None,
                 *args, **kwargs):
        self.fix_order = fix_order
        self.image_width = image_width
        self.image_height = image_height
        super().__init__(*args, **kwargs)

    def _load_annotation(self, sample_id: int) -> dict:
	# The initial annotations.json file contains the format [x_min, y_min, width, height]
        annotation = super()._load_annotation(sample_id)
	# This annotation is in [x1, y1, x2, y2, class_id] format.

        # class label is at the end
        if convert_coco_boxes_to_yolo := True:
            # # This would assume [x, y, w, h]
            # annotation["target"] = CocoDatasetWrapper.convert_my_boxes_to_yolox(
            #             (self.image_width, self.image_height),
            #             annotation["target"])
            annotation["target"] = \
                    CocoDatasetWrapper.convert_absolute_minmaxcorner_box_to_relative_center_box(
                            size=(self.image_width, self.image_height), box=annotation["target"]
                            )

        if self.fix_order:
            # permute each row so that the last entry is now the first.
            # Necessary because https://github.com/Deci-AI/super-gradients/blob/d5a85fd318f4137806c37d73120f905e0d51f6a7/src/super_gradients/training/datasets/detection_datasets/coco_format_detection.py#L144 is wrong (inconsistent with the rest of their pipeline).
            #   E.g. calling dataset.plot() expects the class label to be at the end (index 4),
            #   but the loss expects it at index 0.
            # But then this will cause postprocessings to fail....
            annotation["target"][...,(0,1,2,3,4)] = annotation["target"][...,(4,0,1,2,3)]
            # class label should now be at the start, not at the end

        print(f"{annotation=}")
        return annotation

    @staticmethod
    def convert_absolute_minmaxcorner_box_to_relative_center_box(size, box):
        """
        input box: (xmin_absolute, ymin_absolute, xmax_absolute, ymax_absolute)
        output box: (xcenter_relative, ycenter_relative, width_relative, height_relative)
        """
        # https://stackoverflow.com/a/56121386/2550406
        dw = 1./size[0]
        dh = 1./size[1]
        x = (box[...,0] + box[...,2])/2.0
        y = (box[...,1] + box[...,3])/2.0
        w = box[...,2] - box[...,0]
        h = box[...,3] - box[...,1]
        x = x*dw
        w = w*dw
        y = y*dh
        h = h*dh

        box[...,0] = x
        box[...,1] = y
        box[...,2] = w
        box[...,3] = h
        return box

Above code goes from a COCO json dataset with x_min, y_min, width, height in pixels to a format with the same in relative values in [0,1]. Note that the COCOFormatDetectionDataset transformed that to x_min, y_min, x_max, y_max before we get access to it.
Additionally, my code snippet takes the original COCOFormatDetectionDataset's output and changes the order from (box, class) to (class, box), which is what the yolox loss expects.
Finally, if you want to visualize your results using the DetectionVisualizationCallback or the ExtremeBatchDetectionVisualizationCallback, you will run into a crash. The reason is that they both use this function which expects [class_id, x1, y1, x2, y2] but receives [image_id_in_batch, class_id, x1, y1, x2, y2] for the targets. So you need to patch that file in your installation to look like this:

            for label_xyxy in target_boxes:
                print(f"detection_utils.py: Target box i/?: {label_xyxy}")
                # The callback might give us [image_id, class, xyxy_box]
                # But we want [class, xyxy_box]
                label_xyxy = label_xyxy[-5:]

                image_with_targets = DetectionVisualization.draw_box_title(

I believe the main reason why the super_gradients.training.dataloaders.dataloaders.coco_detection_yolo_format_train behaved better for me is that I had specified a transform for it. There are a lot of different formats in use in this codebase.

The coco_detection_yolo_format_train is a wrapper to the YoloDarknetFormatDetectionDataset which has a conversion from LABEL_CXCYWH to XYXY_LABEL

We can browse all the possible formats at https://github.com/Deci-AI/super-gradients/blob/56de963ef40cd0e4e1c437b90536735ce7f71ba3/src/super_gradients/training/datasets/data_formats/default_formats.py#L12 (and their references in the github sidebar)

Notably, there is also a LABEL_NORMALIZED_CXCYWH. This implies that the dataset actually expects absolute pixel coordinates, although YOLO-format would normally have them normalized. But because the XYXY_LABEL output is also not normalized, this just means that a normalized yolo-formatted dataset on disk will be loaded in normalized [x1 y1 x2 y2 class] format now.

You can specify transforms in the dataset.dataset.transforms = [] property. These allow you to avoid creating a dataset wrapper like I did. For example:
   val_data = coco_detection_yolo_format_val( ... )
   from super_gradients.training.datasets.data_formats.default_formats import XYXY_LABEL, LABEL_XYXY, \
   from super_gradients.training.transforms.transforms import DetectionTargetsFormatTransform
   val_data.dataset.transforms = [ DetectionTargetsFormatTransform(input_size, input_format = XYXY_LABEL, output_format = LABEL_CXCYWH) ]

The distinction between LABEL_XYXY and XYXY_LABEL is whether the class comes first or last. The YOLOX Model expects the class label before the box coordinates. Hence the use of LABEL_CXCYWH.

The model outputs some more complex raw format. This is later converted inside yolox_loss.py#prepare_predictions to some format that can be compared with the targets. I haven't understood this part yet.

A similar (hopefully identical) conversion happens in the Visualization Callbacks, which have a post_prediction_callback that you can set to an instance of YoloXPostPredictionCallback to do this.

After the post-prediction callback, the DetectionVisualizationCallback (and ExtremeBatchDetectionVisualizationCallback) seem to expect predictions to be in absolute XYXY format, and targets in relative CXCYWH format...
This confuses me, so I added the patch at the end of my previous comment, so it also works with absolute values. But I think that should not be necessary if you manage to use the correct data transform at the start.

Finally, I concluded from this that if I transform the initial dataset to LABEL_NORMALIZED_CXCYWH, the model would learn to predict relative coordinates for the bounding boxes. That, however, causes new issues that I don't want to investigate.
I am still using the patches I mentioned in my previous comment.
Hope this helps someone else in the future.

@lucidBrot thanks for you answers.

@mariannaparzych indeed, @lucidBrot is right.
The YoloX loss expects the LABEL_CXCYWH format. If you had anything else in the first index this can explain the error that you had when trying to compute it.
We will update the docs of the YoloX loss so it would be more clear.
I am closing this issue for now, if this or any other problem persists please feel free to re-open this issue or open a new one.


<EMAIL_ADDRESS>extract_dir error

Installing 'azure-cli' (2.0.40) [64bit]
azure-cli-2.0.40.msi (26.0 MB) [==============================================================================] 100%
Checking hash of azure-cli-2.0.40.msi... ok.
Extracting... DEBUG
-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Monday, July 9, 2018 11:23:24 AM
   Source - C:\Users\hpierson\scoop\apps\azure-cli\2.0.40\_tmp\Microsoft\
     Dest - C:\Users\hpierson\SDKs\Azure\CLI2\

    Files :
  Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------------

ERROR : Invalid Parameter #3 : "C:\Users\hpierson\scoop\apps\azure-cli\2.0.40"

       Simple Usage :: ROBOCOPY source destination /MIR

             source :: Source Directory (drive:\path or \\server\share\path).
        destination :: Destination Dir  (drive:\path or \\server\share\path).
               /MIR :: Mirror a complete directory tree.

    For more usage information run ROBOCOPY /?

****  /MIR can DELETE files as well as copy them !

ERROR Could not find 'CLI2'! (error 16)

Please create a new issue by using the following link and paste your console output:
https://github.com/lukesampson/scoop-extras/issues/new?title=azure-cli%402.0.40%3a+extract_dir+error
True

I fixed a bug regarding robocopy with https://github.com/lukesampson/scoop/commit/360c78996508bf5f4ccab26a70240b05ca00d801
Can you run scoop update and try it again? Maybe that was the problem.

Yup, running scoop update brought in https://github.com/lukesampson/scoop/commit/360c78996508bf5f4ccab26a70240b05ca00d801 which appears to have solved the issue. Thanks!


Optimize staticData

Reduce bundle size and runtime memory usage.

Do you have any results on how it impacts the bundle size? I've tried the exact same and it turned out to not safe anything for gzipped comparison.

Actually I didn't test gzipped bundle size (it saves ~1 KB before gzipped). This PR is mainly about reducing runtime memory usage.


Package Manager Error

Describe the bug

When clicking the Package Manager, I encountered this issue.
"Could not load component list:Error https://schemas.botframework.com/schemas/component/v1.0/component.schema: self signed certificate in certificate chain Error Microsoft.QnAMakerDialog: Cannot extend component because it is not included Error Microsoft.QnAMakerDialog: Cannot set property 'const' of undefined *** Could not merge components ***"
Version

Version:  2.1.0-nightly.266659.7c14b7a
Electron: 8.2.4
Chrome: 80.0.3987.165
NodeJS: 12.13.0
V8: 8.0.426.27-electron.0
Browser

[ ] Electron distribution
[x] Chrome
[ ] Safari
[ ] Firefox
[ ] Edge

OS

[ ] macOS
[x] Windows
[ ] Ubuntu

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior

Screenshots

Additional context

@lapulgaatomica10 Did you install or uninstall any components, or does this happen with a brand new bot?
Can you reproduce the problem with a brand new empty bot?

@benbrown yes I tried install and uninstall the composer and after that I tried it on a brand new bot and it is still the same. Should I  uninstall and install the pre-requesites of composer and try it again?

Im getting same issue with brand new QnA Maker template. Navigating to the Package Manager gives this error.
Could not load component list:Error https://schemas.botframework.com/schemas/component/v1.0/component.schema: self signed certificate in certificate chain Error Microsoft.QnAMakerDialog: Cannot extend component because it is not included Error Microsoft.QnAMakerDialog: Cannot set property 'const' of undefined *** Could not merge components ***

@benbrown Ping. Do you think it's an installation issue?

Hi @benbrown how can I know if its network problems? I mean where should I check?

@lapulgaatomica10 Are you on a restricted network, or do you access the internet normally through a proxy? The message about the self signed certificate in certificate chain might indicate that the request for the schema file is being blocked and redirected and thus failing the SSL check.

I have the same issue, I have verified and we have implemented Netscope in our company. I need de the URL's that should be added to the exception rules. Please help me wiht these.

I have exactly the same issue. Is there a solution?


Drop support for anything but stable, beta, (nightly)

libc's CI only tests against stable, beta and nighlty (and a single build seems to run against 1.0.0 for some reason). Since we are a wrapper for libc, I would like to drop support for anything but stable, beta (and working towards getting nighlty green).
If others are not comfortable with this, I would suggest to altenatively only support the n latest releases (in addition to beta and nightly), where n has to be clearly defined.

Closed in favour of #444.


fix(color-mode): use color.preference & use select

Use colorMode.preference as colorMode.value is meant to be read-only. https://color-mode.nuxtjs.org/#usage
Use a <select> instead of a button to add system option

I personally prefer to keep it as a toggle for simplicity. The trick is that we could check if the user color is matched with the system color, if so we could reset it to system. For example:
https://github.com/nuxt/ui-pro/blob/cbbfed17a91aad3bb4fc0288eeca75567058552a/components/color-mode/ColorModeButton.vue#L29-L36


[ICON] arrow with box

Icon Request

Icon name: arrow with box
Use case: login or logout
Screenshots of similar icons:

These icons already exist. Take a look at log-in, log-out, and log-denied.


Support tabs / MDI

Would be handy to be able to have multiple Markdown files open in one window.

i was more in favor of the one-window-one-file paradigm but if you give me compelling enough reasons to have tabs then that might be added :smile:


Validating Fixed and Pattern Values

https://github.com/medplum/medplum/issues/2059

coverage: 94.073% (-0.06%) from 94.128% when pulling 05c0c86a540a4df0457a1a5143ee9f76f071f2c5 on james-fixed-pattern into 10b8d3587ea0614a2556b8db432962a7a4e79d09 on main.


Gather data

Using the TMBD Api key and the Google trends gather the required data. Make sure to save any files that contain data in the /data directory.

The google trends did not work and we were not able to gather data from it, so we are using just the TMBD data


Update n_photos property name to mz_n_photos

@zerebubuth: could you review please?

Merging this in to make the dev deploy easier. @zerebubuth if you find something wrong with it we can update afterwards.

:+1: all looks good to me.


Improve usability of lengthCompare

In most codebases I've seen people just use seq.length/size <> n even though seq.lengthCompare(n) <> 0 is recommended for sequential collections. In some cases it's probably for lack of knowledge, but in others the latter is just deemed too unreadable.
My suggestion is to add some shortcuts that make it more pleasant to use.
Something along those lines:
def lengthEq(n: Int):  Boolean = lengthCompare(n) == 0
def lengthMin(n: Int): Boolean = lengthCompare(n) >= 0
def lengthMax(n: Int): Boolean = lengthCompare(n) <= 0

Other suggestions also welcome, of course.
This particular issue came up on Gitter.

What about this kind of interface?
  implicit class SeqLikeOps[+A, +Repr](s: SeqLike[A, Repr]) {
    object lazyLengh {
      def <(value:  Int) = s.lengthCompare(value) < 0
      def >(value:  Int) = s.lengthCompare(value) > 0
      def ==(value:  Int) = s.lengthCompare(value) == 0
      def <=(value:  Int) = s.lengthCompare(value) <= 0
      def >=(value:  Int) = s.lengthCompare(value) >= 0
    }
  }

That allows to use:
val s: Seq[String] = ???
s.lazyLength < 10
s.lazyLength == 10
s.lazyLength > 10

However, i am not a DSL expert maybe this can be implemented without an extra inner object?

No, but the usability is almost the same (modulo _)

Oh is this suggestion doing the exact kind of thing? sweat_smile #338 (comment)

lengthEquals, lengthMoreThan, lengthLessThan

I like Sébastien's suggestion best.

I guess Sébastien is https://github.com/sjrd and his suggestion is

length_==, length_>=, length_<=

I personally find that syntax a bit ugly. In my opinion we should avoid partially symbolic method names. Do we have examples of similar style elsewhere in the project?
I preferred aleks-sidorenko's lengthEquals, lengthMoreThan, lengthLessThan because it captures the same idea but looks cleaner (albeit slightly more verbose).
There is some elegance in isaias-b's suggestion too, but I think there is benefit with the fact all the methods related to lengthCompare start with the length prefix.

I have actually thought of a way to do @isaias-b's suggestion without having to allocate an object.
trait SeqOps[+A, +CC[_], +C] extends Any
  with IterableOps[A, CC, C] { self =>
  /* ... */

  // open to bikeshedding this name
  def lengthCompare: SeqOps.LengthCompareOps[A, CC, C] = new LengthCompareOps(this)
}

object SeqOps {
  // open to bikeshedding the equals and not equals methods
  final class LengthCompareOps[A, CC[_], C, S <: SeqOps[A, CC, C]] private(val seq: S) extends AnyVal {
    def <(len: Int): Int = seq.lengthCompare(len) < 0
    def <=(len: Int): Int = seq.lengthCompare(len) <= 0
    def ===(len: Int): Int = seq.lengthCompare(len) == 0
    def =!=(len: Int): Int = seq.lengthCompare(len) != 0
    def >=(len: Int): Int = seq.lengthCompare(len) >= 0
    def >(len: Int): Int = seq.lengthCompare(len) > 0
  }
}

Thoughts?

How often do you need lengthCompare(n) >= 0 or lengthCompare(n) <= 0? I would imagine lengthCompare(n) == 0 is by far the most common use case.
How about simply implementing lengthEquals method for now?

I like symbols more than practically everyone (I always use /: for instance), but the alphanumeric-symbolic hybrid suggested here seems really unfriendly to me.
I think lengthAtLeast and lengthAtMost work slightly better than lengthLessThan and lengthMoreThan because of overflow (lengthLessThan(n+1) is bad news when n is Int.MaxValue).
The other option to consider is takeLength(n: Int): Int which returns the exact length unless the length is n or more, at which point it stops at n.

Do we have examples of similar style elsewhere in the project?

We have List.reverse_:::, which is named as such because xs reverse_::: ys is semantically equivalent to xs.reverse ::: ys, but faster.
Likewise, xs length_> 2 would be semantically equivalent to xs.length > 2, but faster.
So we have a very very close precedent for my suggestion.

reverse_::: is only named as such to get right-to-left evaluation, isn't it?

No that's not necessary. It could have been called reverseAndPreprendTo and the implementation would just have swapped this and that.


Application Gateway with AKS fails with muliple backend pods app with Azure AAD authentication

I am using Application Gateway as ingress controller on my AKS cluster https://github.com/Azure/application-gateway-kubernetes-ingress.

Developed a ASP.net Core application with Azure AAD application.
Deployed to AKS cluster
Created Application Gateway ingress controller to access this application via App Gateway URL
All works fine when there is only one pod of application running in AKS.
Things break when scale the AKS application to 2 or more pods, which is a requirement in production case. (for high Availability)
The same app works when Accessed form public IP exposed by AKS (Load Balancer type service)
An unhandled exception occurred while processing the request.
Exception: Correlation failed.
Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler+d__12.MoveNext()
Stack Query Cookies Headers
Exception: Correlation failed.
Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler+d__12.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
System.Runtime.CompilerServices.TaskAwaiter.GetResult()
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+d__6.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Microsoft.AspNetCore.Session.SessionMiddleware+d__9.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
Microsoft.AspNetCore.Session.SessionMiddleware+d__9.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+d__7.MoveNext()
Show raw exception details
System.Exception: Correlation failed.
at Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler1.d__12.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Session.SessionMiddleware.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Session.SessionMiddleware.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.d__7.MoveNext()

I am using Application Gateway as ingress controller on my AKS cluster https://github.com/Azure/application-gateway-kubernetes-ingress.

Developed a ASP.net Core application with Azure AAD application.
Deployed to AKS cluster
Created Application Gateway ingress controller to access this application via App Gateway URL
All works fine when there is only one pod of application running in AKS.
Things break when scale the AKS application to 2 or more pods, which is a requirement in production case. (for high Availability)
The same app works fine with multiple pods when accessed from public IP exposed as AKS load balancer.

An unhandled exception occurred while processing the request.
Exception: Correlation failed.
Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler+d__12.MoveNext()
Stack Query Cookies Headers
Exception: Correlation failed.
Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler+d__12.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
System.Runtime.CompilerServices.TaskAwaiter.GetResult()
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+d__6.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Microsoft.AspNetCore.Session.SessionMiddleware+d__9.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
Microsoft.AspNetCore.Session.SessionMiddleware+d__9.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+d__7.MoveNext()
Show raw exception details
System.Exception: Correlation failed.
at Microsoft.AspNetCore.Authentication.RemoteAuthenticationHandler1.<HandleRequestAsync>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Session.SessionMiddleware.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Session.SessionMiddleware.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.d__7.MoveNext()

@mtuppad why do you thing this is problem with the load-balancer? The exception being thrown seem to be coming from the application pod. It seems to work when only a single instance of the pod is running but seems to fail when the traffic is spread across multiple pods. Also, it would work when you are accessing it from the public IP because the traffic again hits only a single pod.
Could you describe what your application is trying to do? Explaining the architecture and the workflow of the requests would be helpful as well.

Based on offline discussion looks like this is resolved.

@asridharan thanks for closing. Just to clarify, it did work with public ip address (load balancer) when there are more than 2 pods. Its used to fail only when accessed via app gateway.
I am posting solution for others benefit.
Turns out to be dotnet core new api issues. The authentication flow uses data protection API for encryption and decryption of some data (not sure which).  I wasn’t using any shared location to store the keys for the data protection keys. Hence in multi-pod deployment when originating request lands in another pod, issue came. Here is what I did.

Created new storage account in resources group where AKS is deployed. (which is usually Start MC_).  I did try storage account outside of resources group and region, AKS nodes didn’t had access to these storage accounts. SO make sure it’s in same resource group as AKS. Not sure its right thing to as its MC_
Add below code to startup class ConfigureServices method. (sorry couldn't get all code in code block)

`
try
{
string AppStorageAccountConnectionString = Configuration.GetSection("keystore").GetValue("AppStorageAccountConnectionString");
// create a blob storage container to use as the central, cross-node DataProtection key holder
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(AppStorageAccountConnectionString);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference("tupblobsea");
blobContainer.CreateIfNotExistsAsync().GetAwaiter().GetResult();
            // TODO: keys in keys.xml are not encrypted at rest, by default
            // we need to add an encryption cert here so they will be, via .ProtectKeysWithCertificate("certthumbprint")
            services.AddDataProtection(options =>
            {
                options.ApplicationDiscriminator = "todolistwebapp";
            })
            .PersistKeysToAzureBlobStorage(blobContainer, "keys.xml");
        }
        catch (System.Exception ex)
        {

            Console.WriteLine("Error in dataprotection-" + ex.Message);
        }

`

Scaled the app to 4 pods and seems like issue is gone.
Other option is to use volumes to store keys.
https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv
Haven’t tried the file share option, but looks like it should work as all pods will mount same file share.

Some references I used are
https://github.com/IdentityServer/IdentityServer4/issues/720
https://www.kevindockx.com/solving-correlation-failed-state-property-not-found-errors-openid-connect-middleware-asp-net-core/
https://github.com/aspnet/Security/issues/1755#issuecomment-392803709


What are the permissions needed to use this package with confluence server?

Hi.
I'm trying to create a documentation page, but upon running make confluence I get this error:
---
Permission denied on Confluence (REST GET)

The configured user does not have permission to perform an action on the
Confluence instance.

I can't find any logging options to check in detail what is exactly happing. I have the permissions to create pages in the page I've put in conf.py, but apparently confluencebuilder can't do that. Any ideas how to debug this? I'm suspecting maybe this package needs more than just a user-access for documentations.

A permission denied error occurs when the access point returns a 403 error. This could either be the Confluence instance reporting the user does not have permission or a client is not properly authenticated with a (reverse) proxy.
Typically, a user would want Confluence permissions for a specific space to have "Add" permission for pages/attachments; and possibly "Delete" permissions, if supporting content purging. These are typically enabled in default setups, but it depends on how a Confluence administrator configures a given space.
Speculating here, but since the error message indicates "Permission denied on Confluence (REST GET)", it looks to be a first-read (GET) request, attempting to query the instance. If you are using the same user which you can also navigate/create/edit pages with using a browser, this may be an authentication issue with the environment invoking the Sphinx publication request. For example, if you have a server URL configured as follows:
confluence_server_url = 'https://intranet-wiki.example.com/'

And in a browser you can navigate to:
https://intranet-wiki.example.com/confluence/rest/api/search

And see a response such as the following:
{"statusCode":400,"data":{"authorized":false,"valid":true,"errors":[],"successful":false},"message":"com.atlassian.confluence.api.service.exceptions.BadRequestException: cql query parameter is required"}

The URL is fine. And if you try the same REST call on your shell (using curl or wget):
curl https://intranet-wiki.example.com/confluence/rest/api/search

And see a different result, you may require additional options (such as confluence_server_auth or confluence_server_cookies).

One approach could be to enable all Requests logging inside a Sphinx configuration (conf.py) using something like the following:
import logging
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
rlog = logging.getLogger('requests.packages.urllib3')
rlog.setLevel(logging.DEBUG)

And as you run the publishing stage, each request should be dumped to the standard output stream.

If it fails at a later point, it could point to an issue with a specific page/request being made. If it happens on the first request, it is most like a setup/authentication issue as mentioned above.

A permission denied error occurs when the access point returns a 403 error. This could either be the Confluence instance reporting the user does not have permission or a client is not properly authenticated with a (reverse) proxy.
Typically, a user would want Confluence permissions for a specific space to have "Add" permission for pages/attachments; and possibly "Delete" permissions, if supporting content purging. These are typically enabled in default setups, but it depends on how a Confluence administrator configures a given space.
Speculating here, but since the error message indicates "Permission denied on Confluence (REST GET)", it looks to be a first-read (GET) request, attempting to query the instance. If you are using the same user which you can also navigate/create/edit pages with using a browser, this may be an authentication issue with the environment invoking the Sphinx publication request. For example, if you have a server URL configured as follows:
confluence_server_url = 'https://intranet-wiki.example.com/'

And in a browser you can navigate to:
https://intranet-wiki.example.com/confluence/rest/api/search

And see a response such as the following:
{"statusCode":400,"data":{"authorized":false,"valid":true,"errors":[],"successful":false},"message":"com.atlassian.confluence.api.service.exceptions.BadRequestException: cql query parameter is required"}

The URL is fine. And if you try the same REST call on your shell (using curl or wget):
curl https://intranet-wiki.example.com/confluence/rest/api/search

And see a different result, you may require additional options (such as confluence_server_auth or confluence_server_cookies).

One approach could be to enable all Requests logging inside a Sphinx configuration (conf.py) using something like the following:
import logging
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
rlog = logging.getLogger('requests.packages.urllib3')
rlog.setLevel(logging.DEBUG)

And as you run the publishing stage, each request should be dumped to the standard output stream.

If it fails at a later point, it could point to an issue with a specific page/request being made. If it happens on the first request, it is most like a setup/authentication issue as mentioned above.

I have the same problem:
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): confluence.foo.bar:443
DEBUG:urllib3.connectionpool:https://confluence.foo.bar:443 "GET /rest/api/space?limit=1&spaceKey=C2330 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://confluence.foo.bar:443 "GET /rest/api/search?limit=2&cql=type%3Dspace+and+space.title~%22C2330%22 HTTP/1.1" 403 None

However I  wonder what that second request is...? The first one seems to try and get the correct Confluence Space (C2330). The second...?

I have the same problem:
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): confluence.foo.bar:443
DEBUG:urllib3.connectionpool:https://confluence.foo.bar:443 "GET /rest/api/space?limit=1&spaceKey=C2330 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://confluence.foo.bar:443 "GET /rest/api/search?limit=2&cql=type%3Dspace+and+space.title~%22C2330%22 HTTP/1.1" 403 None

However I wonder what that second request is...? The first one seems to try and get the correct Confluence Space (C2330). The second...?

The second GET request is invoked in a situation where the original space key cannot be found. The intent of this feature was to help find/report a space with a similar name, since it has been reported a couple of times where users would use a difference space key casing/etc. and wonder why it couldn't find their space.
In this case, while an success (200) REST call was performed on the Confluence instance, the successful response here is most likely that the Confluence reported that "hey, the request is fine, but I have no space with that name". This is either that the space key does not exist (which I assume is not the case) or that the user requesting a search does not have permission to know that such a space key exists. I recall seeing in the past that anonymous/"unauthorized" requests would have a 200-reported search call with zero results, but a more explicit search can result in a hard authentication failure (403).
In all cases I have seen, a 403 reported error code is solely a result of an authentication issue -- either with the credentials being used to authenticate, or a result of a additional access layer added for Confluence non-Cloud in some corporate environments.

@jdknight thanks for the response! I raised an issue with out IT department, and they changed something -- and suddenly it works again. :) Sadly I don't know what they changed, but afterwards confluencebuilder was able to upload again. Great piece of software!

It is assumed based of the feedback and duration since these events have occurred, these issues have either be resolved or are no longer issue. With that in mind, marking these issues as closed. If this is incorrect, please mention so and we can re-open this issue.


Let maven access the .git directory

We use  the maven commit-id plugin to push git version information inside every build so that build can be traced back to commit-id. This plugin takes git metadata from .git directory and creates a files that's inserted into the build. However, buildpacks apparently does not keep the .git directory visible/accessible to the build process and hence the maven plugin fails to work. Leading to the error:
Failed to execute goal pl.project13.maven:git-commit-id-plugin:4.9.10:revision (default) on project fooservice: .git directory is not found! Please specify a valid [dotGitDirectory] in your pom.xml
Note: We use kpack for build
Describe the Enhancement

It would be nice if the .git directory remains accessible by the build process. Many build scripts would expect that directory to be present for a similar requirement.
Possible Solution

Please let the .git directory be accessible to the build process. If that can't be made the default, provide a flag to do so.
Motivation

Baking source commit-id info into a build is a common practice. It helps in debugging/reporting issues. I read about the git build-pack. It puts commit id into an environment variable. But I can't use that as I need a git.proerties file that the above plugin generates. This file is later used by Spring (framework) at runtime to present build information at a specific http endpoint.

Not sure this is exactly what you want, but have you tried using:

$BP_INCLUDE_FILES | Colon separated list of glob patterns to match source files. Any matched file will be retained in the final image. Defaults to `` (i.e. nothing).
$BP_EXCLUDE_FILES | Colon separated list of glob patterns to match source files. Any matched file will be specifically removed from the final image. If include patterns are also specified, then they are applied first and exclude patterns can be used to further reduce the fileset.

The default behavior is to remove all of the files not necessary from the final image, so after build we delete everything except the compiled assets. Prior to that, during the build, all of your source code and everything that was included with the build should be present. I know when I do a pack build, the .git files are there. I don't know about kpack, maybe it has specific behavior here. Do things work correctly if you do a pack build of your app?

I don't want to retain any extra files after the build. I want the .git directory to be present till the build finishes so that the build process can access data from it. Problem is, the .git directory is missing so the build fails. Buildpacks only presents the source code to the build process and not the .git directory.
About .git directory: When you checkout a git project, you get a .git directory at the base of the project directory. The maven git-commit-id-plugin is supposed to read git revision information from this .git directory and then write a summary to git.properties.

If .git isn't present, it's probably not a buildpack issue. We don't delete files from /workspace until after the build finishes. As previously mentioned, I did a pack build and confirmed I could see .git files. Can you try a pack build and see if that works for you? If it does, then you're probably hitting an issue with kpack and not the buildpack.


Client insecure mode

Added TLS client insecure mode for etcd(v2) backend to mimic "curl --insecure". Allow connections to secured etcd without cert verification.
Enable client insecure mode using -client-insecure or CONFD_CLIENT_INSECURE=true options.

@PascalChardon thanks for opening a PR!
I'm not 100% sure about adding a special environment variable just for that configuration option. I think that support for configuration from environment variables is something that confd lacks, but it needs a generic solution and not a one-off thing. Otherwise the PR looks good. I'll merge if you will remove the part that handles the environment variable.

Ok, You're the boss :-) I'll remove the part with env variable and submit again.

Hi Oleksandr,
I removed the code around env variable and had to fix a small issue on a peace of code I pulled with a merge. Unclear how this code can be in the master branch.

Hi @okushchenko I am in need of this feature. I was able to make build and consume confd binary with -client-insecure
But, when is 0.17.0 going to be released?


Improved and corrected some String code comments

Ported https://github.com/dotnet/coreclr/pull/1987 to CoreRT.
Added comment to the new FastCompareStringHelper.
Removed TODO comment from Compare. We have already done the null checks that CompareOrdinal has so it would just be adding an extra function call.
Removed TODO comment from CompareOrdinal. It is done for performance reasons. It doesn't have the alignment reads and when it is known that it is a full string compare it can use the trick of including the null terminator in odd lengths.

Hi @bbowyersmyth, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.
TTYL, DNFBOT;

LGTM. Thanks for porting the fix over!

Sweet, thanks!


Accept bare newlines as well

To support the IETF's general Robustness principle, I would recommend to accept bare newlines as well on parsing. However, I would also understand if you wanted to remain tight on what input you wanted to accept.

The robustness principle is a good one to keep in mind for most cases but I feel that on major/foundational areas of a specification, no compromise should be made because it tends to harm more than help.
I place line delimiting in that category, along with the charset.


Fix issue where partially matching rules effect each others severity levels

NOTE This won't pass the tests until #683 and #685 are fixed.

Currently rule foo will effect foo-bar due to the way we're pairing rules with their files.
Note - Solution seems a bit dirty - so open to suggestions.
Closes #687
DCO 1.1 Signed-off-by: Ben Griffith<EMAIL_ADDRESS>

Coverage increased (+0.002%) to 97.097% when pulling 6d3dabf1d981114633b7d07ccf5950c11f331a31 on bgriffith:feature/fix-search-array into fe467c397906272b326b12277a9af0ea7ce373f1 on sasstools:develop.

Coverage increased (+0.002%) to 97.097% when pulling 6fae9e0f7d844a56a123f3054def32fe0da276fb on bgriffith:feature/fix-search-array into fe467c397906272b326b12277a9af0ea7ce373f1 on sasstools:develop.

Coverage increased (+0.002%) to 97.097% when pulling ef314293a9b2eaafb3eb3c73577af301c61d0acc on bgriffith:feature/fix-search-array into fe467c397906272b326b12277a9af0ea7ce373f1 on sasstools:develop.


Added tpr extension

Reasons for making this change:
After refactoring a *.tmcRefac gets created. Then after a build it gets converted into a *.tpr file. No need to have this file under source control.
Links to documentation supporting these rule changes:
There is no Beckhoff documentation on this file extension.

Have you asked Beckhoff support of whether this file should be handled in version control or not? That's the second best thing to do after checking the documentation in this case I'd say.

@sagatowski  No I haven't yet. So far I haven't noticed issues when removing it. I'll check with support next week.

closing files are needed.

@Roald87
What are the files needed for?

I'm still discussing with support. I will update the stackoverflow question later when I have the full answer.


hello，in https://riverpod.dev/. Can you make a simplified Chinese document for learning?

你好，在in https://riverpod.dev/ 里面，你能做一个简体中文的学习文档吗?
hello，in https://riverpod.dev/. Can you make a simplified Chinese document for learning?

Hello
I don't speak simplified Chinese.
If you want such language, feel free to find someone who could translate the website 😊

hi, i translated the webside in Simplified Chinese whitch on branch docs-v2. #2129
我在docs-v2分支上添加了简中的翻译 #2129


Quest tracker unreadable in Warhammer 4th Edition [BUG]

Module Version: v0.5.1
Describe the bug
I am using the Quest tracker with the Warhammer 4th Edition world and there is an issue in that the text is not readable as the background and text are both dark.
To Reproduce
Steps to reproduce the behavior:

Create a new game using the Warhammer 4th edition world/ruleset
Add the' Forien's quest log' module
Run the game
Try to add a quest to the quest log/view the quest log

Expected behavior
Expect to be able to read the text on the quest log window
Screenshots

Browser:

Chrome 85.0.4183.121 - Mac OS 10.15.6

Foundry Version:
0.6.6
Game System:
Warhammer 4th edition v2.1.0
Additional context
Have tried with and without any other modules installed, is always the same

Fixed: https://github.com/moo-man/WFRP4e-FoundryVTT/releases/tag/3.5.0


cannot install delve - could not import internal/goarch

I raised this on the delve help forum
https://github.com/go-delve/delve/issues/3317
and they redirected me here
Hi all, probably doing something stupid, but this is what I tried:
What version of Go are you using (go version)?
go version
go version go1.20.2 windows/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
New build i9 13900K Windows 10: 10.0.19045
32GB RAM, tons of hard drive space.
This is a new PC and hasn't had GO installed on it before.
go env Output
go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\conta\AppData\Local\go-build
set GOENV=C:\Users\conta\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\conta\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\conta\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20.2
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=0
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\conta\AppData\Local\Temp\go-build2941949347=/tmp/go-build -gno-record-gcc-switches

What did you do?
Ran the installer go1.20.2.windows-amd64
Checked version:
go version
go version go1.20.2 windows/amd64

Tried to install delve
go clean -modcache
go install github.com/go-delve/delve/cmd/dlv@latest

What did you expect to see?
Delve install OK
What did you see instead?
could not import internal/goarch (EOF) and various others...
go: downloading github.com/go-delve/delve v1.20.1
go: downloading github.com/sirupsen/logrus v1.6.0
go: downloading github.com/spf13/cobra v1.1.3
go: downloading github.com/google/go-dap v0.6.0
go: downloading github.com/mattn/go-isatty v0.0.3
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/cosiner/argv v0.1.0
go: downloading github.com/go-delve/liner v1.2.3-0.20220127212407-d32d89dd2a5d
go: downloading github.com/mattn/go-colorable v0.0.9
go: downloading github.com/derekparker/trie v0.0.0-20200317170641-1fdf38b7b0e9
go: downloading golang.org/x/sys v0.0.0-20220908164124-27713097b956
go: downloading golang.org/x/arch v0.0.0-20190927153633-4e8777c89be4
go: downloading github.com/hashicorp/golang-lru v0.5.4
go: downloading go.starlark.net v0.0.0-20220816155156-cfacd8902214
go: downloading github.com/cilium/ebpf v0.7.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.0
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/mattn/go-runewidth v0.0.13
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.3
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading github.com/shurcooL/sanitized_anchor_name v1.0.0
# runtime/internal/math
C:\Program Files\Go\src\runtime\internal\math\math.go:7:8: could not import internal/goarch (EOF)
# runtime/internal/sys
C:\Program Files\Go\src\runtime\internal\sys\consts.go:8:2: could not import internal/goarch (EOF)
C:\Program Files\Go\src\runtime\internal\sys\consts.go:9:2: could not import internal/goos (EOF)
# github.com/shurcooL/sanitized_anchor_name
go\pkg\mod\github.com\shurcoo!l\sanitized_anchor_name@v1.0.0\main.go:10:8: could not import unicode (EOF)
# github.com/rivo/uniseg
go\pkg\mod\github.com\rivo\uniseg@v0.2.0\grapheme.go:3:8: could not import unicode/utf8 (EOF)
# internal/abi
C:\Program Files\Go\src\internal\abi\abi.go:8:2: could not import internal/goarch (EOF)
# internal/bytealg
C:\Program Files\Go\src\internal\bytealg\bytealg.go:8:2: could not import internal/cpu (EOF)
# math
C:\Program Files\Go\src\math\exp_amd64.go:9:8: could not import internal/cpu (EOF)
C:\Program Files\Go\src\math\fma.go:7:8: could not import math/bits (EOF)

looks like a corrupt installation of go

Same problem here, did you find a solution?

I'm so sorry I didn't make a note of the solution. I suspect it was a question of re-installing Golang, but a version that was (fully) supported by VSCode. Currently I am using 1.18.5 so my suggestion would be to try that. It's a pain as VSCode tries to update to latest. Though of course it may be something else that has been fixed/broken in the intervening 10 months. Good luck!


Inconsistency between Ads documentation for billingSetup and ResourceNames.paymentsAccount() method

I'm trying to use the ResourceNames.paymentsAccounts() method to generate the payments accounts resource name for a billing setup create request.
The method requires the customer id, payments profile id, and payments account id.  However, the billing setup create request documentation states that if the payments account id is sent, the payments account info (which contains the payments profile id) should NOT be sent.  The documentation also state that the billingSetups payments_account resource name is customers/{customer_id}/paymentsAccounts/{payments_account_id} and NOT customers/{customer_id}/paymentsAccounts/{payments_profile_id}~{payments_account_id}.
The required parameters for the ResourceNames.paymentsAccounts() method seem to conflict with the documentation.
Which one is correct?

In this case the documentation is correct, and the ResourceNames.payemntsAccount() implementation is wrong. I'll file a PR shortly to address this.

Thanks for the very prompt response.