New clickhouse driver is using a different impl of PreparedStatement

Hi. We noticed a change in behavior when upgraded clickhouse driver and started using com.clickhouse.jdbc.ClickHouseDriver instead of the legacy driver from ru.yandex.*. More specifically, the change is in the way the dates are written into tables. Here are the details:
The new clickhouse-jdbc driver is using a different implementation of PreparedStatement which is InputBasedPreparedStatement whereas the previous version from the legacy driver ru.yandex.* was using SqlBasedPreparedStatement implementation. They have different implementation code in terms of how the datetime values are written into the database. The way we populate the datetime column in our code is as follows:
ps.setLong(i, statistic.getDate().getTime() / 1000L);
So in other words, we take the epoch time, remove millisecond part and let it write to the database as is. So the final value would be stored in UTC and when we read it, it would be translated to whatever timezone is set in the server configs or connection string params(we rely on the former).
In the new version of the driver, InputBasedPreparedStatement is using a different approach:

It takes an epoch time which we provide.
It translates it to the DateTime value such as YYYY-MM-DD HH:MM:SS in UTC timezone.
When it is saved to DB, it is treated in the timezone which is set in the server OR the timezone set via connection string params.
Since we have America/Toronto set in the server, the DateTime value which was converted to UTC is treated in the America/Toronto timezone.
Once it is saved on the server side, it would be converted again to UTC, i.e., DateTime value + 5h.

As a result of that, we see all stats shifted by 5 hours ahead. We found that in order to remediate this issue, we have to set the following connection string params:
jdbc:clickhouse://<IP_ADDRESS>:18123/ssdsp?use_server_time_zone=false&use_time_zone=UTC
Now the questions. Why does the new driver use InputBasedPreparedStatement and not SqlBasedPreparedStatement anymore? Is there a better way to handle this? Unfortunately, I wasn't able to find any comments in the release notes relating to that change. Please advise.

Hi @malikas05, thanks for the report with all the details.

Why does the new driver use InputBasedPreparedStatement and not SqlBasedPreparedStatement anymore?

The driver was refactored in 0.3.2 to resolve stability issue and the data format was changed from TabSeparated to RowBinary for better performance. InputBasedPreparedStatement is good in most cases because it can validate inputs and send binary data to ClickHouse, while SqlBasedPreparedStatement is nothing but a SQL query with all parameters substituted, which is less efficient(not only on client side but also server).
If you prefer SqlBasedPreparedStatement, before a new option(e.g. auto_type_inferring=false) in place, you may add custom expression like insert into mytable values(ifnull(?, ''), ?) so that the driver will give up type inferring.

Is there a better way to handle this?

Have you tried to pass LocalDateTime object instead to the driver? Alternatively, if you prefer a more direct way, you may try ClickHouseWriter instead.

You're right, using LocalDateTime makes no different :p However, Instant or Timestamp should work - please refer to below.
// server time zone is set to America/Toronto
Properties props = new Properties();
// props.setProperty("use_server_time_zone", "false");
// props.setProperty("use_time_zone", "Etc/UTC");
try (ClickHouseConnection conn = newConnection(props); Statement s = conn.createStatement()) {
    s.execute("drop table if exists test_insert_datetime;"
            + "create table test_insert_datetime(id Int32, ts DateTime) engine=Memory");
    final long epochMilli =<PHONE_NUMBER>123L; // 2023-03-09 00:20:01.123
    final java.sql.Timestamp ts = new Timestamp(epochMilli);
    try (PreparedStatement stmt = conn.prepareStatement(
            "insert into test_insert_datetime")) {
        stmt.setString(1, "1");
        stmt.setLong(2, ts.getTime() / 1000L); // 2023-03-09 00:20:01
        stmt.addBatch();
        stmt.setString(1, "2");
        stmt.setString(2, "2023-03-09 00:20:01"); // 2023-03-09 00:20:01
        stmt.addBatch();
        stmt.setString(1, "3");
        stmt.setObject(2, LocalDateTime.ofEpochSecond(1678321201L, 123000000, ZoneOffset.UTC)); // 2023-03-09 00:20:01
        stmt.addBatch();
        stmt.setString(1, "4");
        stmt.setObject(2, Instant.ofEpochMilli(epochMilli)); // 2023-03-08 19:20:01
        stmt.addBatch();
        stmt.setString(1, "5");
        stmt.setTimestamp(2, ts); // 2023-03-08 19:20:01
        stmt.addBatch();
        stmt.executeBatch();

        ResultSet rs = s.executeQuery("select * from test_insert_datetime order by id");
        Object[][] values = new Object[5][];
        int index = 0;
        while (rs.next()) {
            values[index++] = new Object[] { rs.getInt(1), rs.getObject(2) };
        }
    }
}

Although we have workaround here, it's indeed inconsistent with clickhouse-client, the native command line:
# start with UTC
docker run --rm -it -e TZ=Etc/UTC clickhouse/clickhouse-server bash
./entrypoint.sh >/dev/null 2>&1 &
clickhouse-client
...
select timezone(), toDateTime(1678321201)
...
┌─timezone()─┬─toDateTime(1678321201)─┐
│ Etc/UTC    │    2023-03-09 00:20:01 │
└────────────┴────────────────────────┘

# now let's try your timezone
docker run --rm -it -e TZ=America/Toronto clickhouse/clickhouse-server bash
./entrypoint.sh >/dev/null 2>&1 &
...
select timezone(), toDateTime(1678321201)
...
# use server timezone
clickhouse-client
┌─timezone()──────┬─toDateTime(1678321201)─┐
│ America/Toronto │    2023-03-08 19:20:01 │
└─────────────────┴────────────────────────┘
...
\q
# use client timezone
TZ=Etc/UTC clickhouse-client --use_client_time_zone 1
select timezone(), toDateTime(1678321201)
...
┌─timezone()──────┬─toDateTime(1678321201)─┐
│ America/Toronto │    2023-03-09 00:20:01 │
└─────────────────┴────────────────────────┘

Perhaps we should add a new option to bypass step 2 and 5.
...

It translates it to the DateTime value such as YYYY-MM-DD HH:MM:SS in UTC timezone.
...
Once it is saved on the server side, it would be converted again to UTC, i.e., DateTime value + 5h.


Minor bugfix

Removed requirements of subprocess from setup.
Exception handled if not able to connect to torrent.

thank you for your help.


tsDecorators not support  @inject('appStore') @observer from mobx

  react({
      tsDecorators: true,
    }),

// tsconfig.json
{
  "compilerOptions": {
    "allowJs": true,
    "allowSyntheticDefaultImports": true,
    "baseUrl": ".",
    "checkJs": false,
      // Set true if you see parsing errors in your browser
      "useDefineForClassFields":true,
    "experimentalDecorators": true, // 开启装饰器使用
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "isolatedModules": true,
    "jsx": "react-jsx",
    "lib": [
      "dom",
      "dom.iterable",
      "ES2020"
    ],
    "strictNullChecks":true,
    "module": "ES2020",
    "moduleResolution": "Bundler",
    "noFallthroughCasesInSwitch": true,
    "noEmit": true,
    "noImplicitAny": true,
    "noUnusedLocals": false,
    "paths": {
      "@/*": ["src/*"], // ts 文件使用别名（注释也可用阿，没找到解释）
    },
    "resolveJsonModule": true,
    "skipLibCheck": true,
    "strict": true,
    "target": "ES2022",
  },
  "exclude": [
    "dist",
    "node_modules",
    "public"
  ],
  "include": [
    "./src",
    "**.ts",
    // "**.tsx",
    // "src/**/*.js",
    // "src/**/*.tsx",
  ]
}

// index.jsx
@inject('appStore')
@observer
class Index extends React.Component {
...

tsDecorators are only available in TS(X) files, to limit the usage of unspec decorators in JS files. Given that esbuild shipped spec compliant decorators which is available in Vite 5.3, this should I think fix your issue if you depend on the new stantard (which my not be the case given  "experimentalDecorators": true, in your TS config)

i use mobx:5.15.4 "mobx-react": "6.1.8"  with experimentalDecorators:true
Thank you, I heard that there are several ts specifications. Does swc only support the latest ones? I don't know what kind of converter mobx needs, and does swc have it?
In addition, I have enabled conversion in tsconfig, but it has no effect.
I'll change it to a.tsx file and try again

I've look into it and the support from SWC is not quite there (it can be added by using https://swc.rs/docs/configuration/compilation#jsctransformdecoratorversion), but it not as stable in SWC as in esbuild. I've updated the description of https://github.com/vitejs/vite-plugin-react-swc/issues/86 to reflect this and I'm closing this as duplicate.
For now you can either patch to opt-in to unstable 2022-03 or use the legacy one (both as declared supported by mobx) using tsDecorators option (which requires using TS/TSX source code)


Extend getGlyphMapForStandardFonts with additional entries (issue 15977)

/botio test

I know I'm a bit stubborn sometimes and I already asked the question (https://github.com/mozilla/pdf.js/pull/15034#discussion_r895469505), but I think we should fill the gaps without waiting for bugs.
Based on the pdf you added in this PR, I created this pdf:
0000_11C4.pdf
with all chars code from 0000 to 11C4.
If the final map is too big we can still save it in a binary format (and even compress it since we have stuff to uncompress data).
I have the impress that Chrome is rendering correctly this file (when I compare to Acrobat).

/botio makeref

I know I'm a bit stubborn sometimes and I already asked the question (#15034 (comment)), but I think we should fill the gaps without waiting for bugs.

Maybe, but in that case we probably need to generate the list from a suitable font file since doing that manually doesn't scale beyond a handful of entries at a time.

Based on the pdf you added in this PR, I created this pdf:
0000_11C4.pdf

Just a FYI, if you end up submitting a patch with that file included: That PDF document is corrupt, so you'd probably want to run it through a suitable tool to fix that. (Unless you want to test PDF corruption, it's always nicer if the document is actually valid.)

If the final map is too big we can still save it in a binary format (and even compress it since we have stuff to uncompress data).
I have the impress that Chrome is rendering correctly this file (when I compare to Acrobat).

I don't think the size will be that much of a problem, and note it's already being lazily initialized, so I'd suggest that we don't increase the complexity here.


Reuse datatypes from Ecore in Kolasu

We instead recreate our own

This has been done. We now have these lines:
    val stringDT = EcorePackage.eINSTANCE.eString
    val charDT = EcorePackage.eINSTANCE.eChar
    val intDT = EcorePackage.eINSTANCE.eInt
    val longDT = EcorePackage.eINSTANCE.eLong
    val booleanDT = EcorePackage.eINSTANCE.eBoolean


Fix cocoapods privacy manifest: Set s.resource_bundle for PrivacyInfo…

….xcprivacy in podpsec template

Investigate if this change is needed in the Podspec template:
    spec.cocoapods_version = '>= 1.13.0'


number input should give way to target increment and decrement buttons separately

Currently both have data-part="spin-button".

Currently have to do something like this
[data-part="spin-button"]:first-of-type {
  @apply rounded-l;
}

[data-part="spin-button"]:last-of-type {
  @apply rounded-r;
}


nodejs execution

Hello! Is possibile to use with nodejs in a backend environment? Because i have seen only client-side javascript example.
Thank you

Hiya!  Someone seems to have done this in #25
Please let me know if it works for you.  I'm hoping to add a Wiki section on this, and your experience would be most helpful.
Gonna close this ticket, so update us over at #25 if you can 👍


Link for further information about gate pluggable authentication WORTHLESS

The link supplied for the text below gets redirected to a general support page and cannot search/find "paa-sample". Update this link so this page is more useful.
"Custom authentication and authorization models before Windows 8.1 are not supported, although the article above discusses them.
To learn more about this feature, check out https://aka.ms/paa-sample."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0517ea6a-9417-31a8-39a5-f6128387fee5
Version Independent ID: 7636ec70-e1dd-8c6b-516c-8af324d9f645
Content: Get started with the macOS client
Content Source: WindowsServerDocs/remote/remote-desktop-services/clients/remote-desktop-mac.md
Product: windows-server
Technology: remote-desktop-services
GitHub Login: @lizap
Microsoft Alias: elizapo

I wish I could contribute with something useful here. I struggle at finding the original source for that link, but I am not sure if that is because I am not much good at C# coding or programming in general.
Starting points to continue the search from:

https://support.microsoft.com/en-us/help/2830477/update-for-remoteapp-and-desktop-connections-feature-is-available-for
Bing search "pluggable authentication and authorization" site:microsoft.com: https://www.bing.com/search?q="pluggable+authentication+and+authorization"+site%3Amicrosoft.com&form=QBLH&sp=-1&pq="pluggable+authentication+and+authorization"+site%3Amicrosoft.com&sc=0-63&qs=n&sk=&cvid=1F4DC12F9D49419A8FB3FD0A953E7861

#assign:HeidiLohr

We hugely appreciate your contribution, feedback, and support to our technical documentation.
We’ve phased out GitHub Issues on this repository as the feedback mechanism for content and have replaced it with a new feedback system. For more information see, Content User Feedback. As part of this rollout, we are migrating your issue where our team will use your comments to improve the documentation experience. Unfortunately, this does mean you won't be able to access this issue after the migration is complete.
All issues will be migrated by Thursday 29th August 2024.
Once again, thank you for your valuable feedback, we hope you’ll continue to contribute using our new feedback system.


How to use behind proxy that require authentication?

Minimal repro steps
Run the following command in console
libman install bootstrap -d wwwroot/lib/bootstrap --files dist/css/bootstrap.css --files dist/css/bootstrap.min.css --files dist/js/bootstrap.js --files dist/js/bootstrap.min.js
Expected result
Download the library files
Actual result
Microsoft.Web.LibraryManager.Contracts.ResourceDownloadException: Failed to download resource from "http://unpkg.com/bootstrap/package.json"
at Microsoft.Web.LibraryManager.WebRequestHandler.GetStreamAsync(String url, CancellationToken cancellationToken)
at Microsoft.Web.LibraryManager.Extensions.GetJsonObjectViaGetAsync(IWebRequestHandler webRequestHandler, String url, CancellationToken cancellationToken)   at Microsoft.Web.LibraryManager.Providers.Unpkg.UnpkgCatalog.GetLatestVersion(String libraryName, Boolean includePreReleases, CancellationToken cancellationToken)
The "bootstrap" library could not be resolved by the "unpkg" provider
Further technical details

I am behind corporate proxy that require  authentication
The solution that I am currently using is activating traffic capture in Fiddler while I am using libman

I'm having the same problem behind a corporate proxy that requires authentication.  I was just on my way over to submit an issue when I saw this one already created.
libman restore produces the same error as @DavoBR submitted.

libman.json : error LIB002: The<EMAIL_ADDRESS>library could not be resolved by the "unpkg" provider

dotnet build causes the same error when Enable Restore Client-Side Libraries on Build is selected.
I'd like to add that running msbuild directly on the project works.  I thought dotnet.exe was built on top of msbuild so I'm not sure why dotnet.exe isn't working too.

I have the same issue. We need a way to specify a proxy server.

We are experiencing this issue with a azure devops/vsts build agent behind a proxy.
I tried to switch to msbuild as @jmcmillen  suggested and that isn't working either.

I have similar problem behind corporate proxy:
when I run libman install jquery
I get error: [LIB002] library could not be resolved by the "cdnjs"

I do have same problem when I am trying to publish ASP.Net core web application to azure container registry.
libman.json : error LIB002: The<EMAIL_ADDRESS>library could not be resolved by the "cdnjs" provider
libman.json : error LIB002: The<EMAIL_ADDRESS>library could not be resolved by the "cdnjs" provider
...

@madskristensen, @raquelsa, @bhsubra, @prafullbhosale, @alexgav, @jimmylewis – any suggestions?

Anyone running into this - can you let me know what responses you're getting, e.g. through Fiddler?  Are you getting HTTP 407 responses from the proxy?
I don't have a proxy here so I'm approaching this from what I can find with my googlebing-fu.

@jimmylewis,
In the command window I run command:
libman install bootstrap
and get message (Russian Windows):
Не удалось скачать каталог от поставщика "cdnjs"
[LIB002]: Поставщику "cdnjs" не удалось разрешить библиотеку "bootstrap"
(Cannot download catalogue from the provider "cdnjs"
[LIB002]: library "bootstrap" could not be resolved by the provider "cdnjs")
In the Fiddler the libman tries to connect:
CONNECT aka.ms:443 HTTP/1.1
and gets response:
HTTP/1.1 407 authenticationrequired

All other programs that enable proxy configuration (e.g. git, vscode, NuGet, my own downloader) work correctly through our company proxy.

In my own downloader (.NET, C#):
using (WebClient wc = new WebClient())
{
WebProxy p = new WebProxy(proxy, true);
p.Credentials = new NetworkCredential(login, password);
WebRequest.DefaultWebProxy = p;
wc.Proxy = p;
// Downloading files.
}
In the code, proxy = "http://proxy:port".

Maybe, we talk about different versions of libman?
I have installed it using command:
dotnet tool install --global Microsoft.Web.LibraryManager.Cli

@jimmylewis Hi!
I have a workaround for the problem:
I cloned the solution and changed the private constructor of the WebRequestHandler class (src\LibraryManager\CacheService\WebRequestHandler.cs):

private WebRequestHandler()
{
    HttpClientHandler httpMessageHandler = new HttpClientHandler();
    httpMessageHandler.Proxy = new WebProxy("http://proxy:port", true);
    httpMessageHandler.Proxy.Credentials = new NetworkCredential("domain\username", "password");
    httpMessageHandler.UseProxy = true;

    _httpClient = new HttpClient(httpMessageHandler);
}

(I also added using System.Net; at the beginning).
I built the solution using the command:
dotnet build
After that in my own project's folder I run the commands:
dotnet "C:\Users\username\SourcePath\LibraryManager\src\libman\bin\Debug\netcoreapp2.1\libman.dll" init
dotnet "C:\Users\username\SourcePath\LibraryManager\src\libman\bin\Debug\netcoreapp2.1\libman.dll" install jquery
or any other external library – twitter-bootstrap, react etc.
And it works just fine. :)
Also interesting fact is that after that I run this command, I also can run the:
libman install jquery
(where libman.exe is installed by running the command from the my previous comment) and it also works pretty good, so I make conclusion that it creates and uses some local cache.
So you can add some command line option or config file that will contain this settings for proxy to resolve the issue.

How about submitting a pull request?

(replying here instead of on linked PR, since this is where the larger discussion is happening)
@Roman-Tarasiuk I assume your proxy uses NTLM authentication?  My PR fix worked for me when I had a proxy using Basic authentication, once I had put the credentials into another app (e.g. the browser), which caused the creds to be stored in the Windows Credential Manager.  I tested it by clearing the LibMan cache (i.e. force it to download files again), building without my PR change, install a package, and verifying that it failed (HTTP 407 - proxy authentication failed), then build with my change, install a package, and it worked.  Changed the proxy password, repeat, same results (once I updated the stored credentials).
I don't have a setup for a proxy with NTLM or a different auth mechanism, so I'm not sure how to test there or how the behavior would be different.  Basic auth seemed to reuse the stored settings in the Credential Manager, but NTLM might be different.  Would love suggestions for how to improve the PR.
Things I'm trying to avoid:

Hard-coding credentials.  Obviously, this won't work from user to user.
Saving credentials to disk.  This is obviously an important security risk.  I can look into this more if we can't figure something else out, but I'd prefer to avoid it; the threat model and mitigations would likely get pretty messy.  It might be possible to use the Credential Manager here too, but there isn't a native .NET API for it, and it is platform specific, and it would require managing prompts whenever credentials change.
Having proxy settings be all-or-nothing.  I could easily imagine a scenario where someone is behind a proxy at work, but takes their laptop out and no longer needs the proxy configuration.  In this case, there should be an easy way to toggle back and forth, instead of having to manually configure LibMan each time.  My PR attempts to use the system settings, so that LibMan isn't managing the proxy settings on its own.  Again, alternative suggestions welcomed; the .NET APIs around proxy configuration seem to be pretty opaque about how to use them properly.

A new version of the CLI package has been published to support this scenario (2.0.48).  It introduces a new libman config command which can be used for proxy settings:
libman config --set http_proxy=http://example will set a proxy server.
For users who need to specify credentials, we currently have a Windows-only solution.  If we get feedback to support this on other platforms we can look into options.  Credentials should be specified as:
libman config --set http_proxy.user=myusername --setEncrypted http_proxy.password=passwordPlaintext
To disable the proxy, you can set it to empty: libman config --set http_proxy=
Lastly, if you have URLs that bypass the proxy, you can use a semicolon delmited list with:
libman config --set http_proxy.bypass=http://foo;http://bar

@jimmylewis, it works pretty good! Thanks a lot!
:)

Almost all my js library are failing to restore when i run restore client side library for my libman file. I am recieving this error " library could not be resolved by the "cdnjs" provider"
example:
{
"library"<EMAIL_ADDRESS>"destination": "wwwroot/lib/twitter-bootstrap/"
},
{
"library"<EMAIL_ADDRESS>"destination": "wwwroot/lib/font-awesome/"
},
{
"library"<EMAIL_ADDRESS>"destination": "wwwroot/lib/toastr.js/"
},
None coud able to restore. I am using visual studio 2019.

I faced the same problem. Libraries are not loaded<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>"bootstrap-select<EMAIL_ADDRESS>

@will056 can you verify that it's an issue with the LibMan tool, and that the provider's CDN is working correctly?  As someone pointed out in a separate issue (but commenting about the same problem), Cdnjs had an outage earlier.  We can't do anything to improve beyond their reliability. 🤔
If it is a problem with LibMan and not the underlying CDN, please open a new issue.  This problem seems unrelated to this current issue.

The issue was with the provider not with Libman tool. CDN had issues yesterday.
https://status.cdnjs.com/incidents/0gdmkm4tl98d. Now all is working fine again.

I'm having an issue, where I physically cannot change the proxy config value. I run an administrator command prompt, run libman config --set http_proxy= and nothing happens. The proxy value is set by default to a proxy server of my workplace, but that's not the correct one that developers use, and I can't change it. Setting the user and password values work, but actually changing the proxy address does not.


Flag kphp2cpp-binary does not properly handle paths with ~

Command
ktest bench --kphp2cpp-binary=~/kphp/objs/bin/kphp2cpp Benchmark.php

Actual behavior
build error: fork/exec ~/kphp/objs/bin/kphp2cpp: no such file or directory

Expected behavior
No error as for:
ktest bench --kphp2cpp-binary=/home/pmakhnev/kphp/objs/bin/kphp2cpp Benchmark.php

~ is expanded by the means of bash or by another shell mechanism.
Bash doesn't expand ~ in some contexts, but it does expand it in this one:
ktest --kphp2cpp-binary ~/path

Notice the use of space instead of =.
You can also use variables if you prefer =.
ktest --kphp2cpp-binary=$HOME/path

So it's not a bug.

I would also suggest setting KPHP_ROOT env var instead of passing --kphp2cpp-binary directly.


Laravel 55 composer updates

PHP and Laravel version have been fixed. The new auto discovering service package feature has been included as well. Updated autoloading to PSR-4.
A few of the packages should be looked at potentially, such as PHPUnit. L55 requires 6.0 now.

I'll close this PR sinsce it a little old and not compatible anymore. If you still need Laravel 5.5 feel free to create a new one.

Just use version 5.1.0, where support for laravel 5.5 was added.


Fix theming of doc background

This essentially makes all #57111 changes completely useless and will cause full RTL context recreation on each resize.  #57111 was more of a workaround, not a proper fix, I still plan to add one optimization to the TextServer, but to do it properly we should have some way to detect what was changed in the theme, instead of always sending NOTIFICATION_THEME_CHANGED.
Anyway, it's probably better to do this in the EditorHelp::_notification -> NOTIFICATION_THEME_CHANGED instead.

Maybe the project_settings_changed signal can be used to react to actual theme configuration changes and not GUI updates due to resize/minsize change.

Maybe the project_settings_changed signal can be used to react to actual theme configuration changes and not GUI updates due to resize/minsize change.

Margin update should be done on resize, since it depends on the control size, but the same function is used for both margin and generic theme changes.

but to do it properly we should have some way to detect what was changed in the theme, instead of always sending NOTIFICATION_THEME_CHANGED

For some reason, I was thinking doing it will require a lot of changes, but seems like it's really easy to do by adding a few lines of code. So I probably will add this, and additional TextServer methods to change font and font size without recreating text buffer on a next Monday/Tuesday. And revert #57111, since it won't be necessary anymore (this should fix this issue as well).

@bruvzg Thanks for pointing that out, I've fixed it by adding an additional parameter to force updating a theme that activates only at NOTIFICATION_THEME_CHANGED.


Nick: Watch lincolnwrites.com

Nick requests the watch of the watch_keyword lincolnwrites\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

lincolnwrites\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by cigien in Charcoal HQ


Create a ROCK for Dex for CKF 1.7

What needs to get done

Create a dedicated repo for dex-rock
Create the rockcraft.yaml file
Copy our CI automation for building the image on every PR and when a PR is merged

Why it needs to get done
In order for us to have our own built OCI Image for that component
Progress

[x] canonical/dex-auth-rocks#1
implement CI:

[x] manually triggered build and publish (canonical/dex-auth-rocks#2)
[ ] automate build and publish CI to trigger on pull request and merge to a protected branch

Created another issue for the ROCK's CI https://github.com/canonical/dex-auth-rocks/issues/4, closing this one as completed.


[Bug][ONNX][EdgeCase] Crash on Scalar Reduce

import torch

class Net(torch.nn.Module):
    def forward(self, x):
        return x.sum()

net = Net().eval()

i = torch.ones((), dtype=torch.float)

with torch.no_grad():
    torch.onnx.export(net, (i), "output.onnx", verbose=True, opset_version=14)

Scalar reduce like the example above is allowed in ONNX and can be executed by ONNXRuntime. But it will crash at tvm::relay::GetReduceAxes in TVM. Same things happens for min/max/mean/sum. I guess this should be an ONNX conversion issue.
cc: @masahi @AndrewZhaoLuo

def @main(%v0: float32) {
  sum(%v0, axis=[])
}

This relay IR crashes when doing type infer.

One quick way to fix it is to check rank of tensor being reduced during conversion:
class Reduce(OnnxOpConverter):
# ...
    @classmethod
    def _impl_v1(cls, inputs, attr, params):
+        if not infer_shape(inputs[0]):
+            return inputs[0]
+
        if "axes" in attr:
            axis = attr.get("axes", 0)
        else:
            axis_len = len(infer_shape(inputs[0]))
            axis = list(range(axis_len))

        return cls.run_calculation(inputs, axis, attr.get("keepdims", True))

But I think it won't be compatible with other reduce operators like ReduceL2 etc.


Resistors other than 1k and 4.7k combination does not work accurately

Sorry i'm pretty new to Arduino and am quite a beginner at coding, so I had plenty of trouble with this. I assumed you could use other values of resistors since the guide i read on Arduino Playground suggested that could be done.
After playing around with other resistance values on 16 key 4x4 keypad because I realize I didn't have 4.7k resistors on hand, I realized 10k + 47k does not work even after changing the values in the sketch,
1k + 2.22k will not read the 4th column accurately. (1st - 3rd column is ok) 220ohm + 330ohm also have accuracy problems.
Please let me know if the problem I found is the fault of my inexperience, or did I really come upon a bug in the calculation?
P.S. my math is horrible and my fundamentals of circuits is horrible so I couldn't really tell what the calculation is about in the library

I added a function to test your resistor values and if the results are not in sequence or are too close to each other, the resistors will not work.

ah, i see. just discovered the ShowRange(); function... it was not in your documentation, sorry I missed it. this is handy!

ok with the help of the function, i found that 1k on rows and 220 on cols can work, and also that the keypad I am using has a short on the pin sometimes, making column 4 wonky. thanks for the help, and sorry for the trouble!


Query cloudformation for template when starting cluster

The start command does not allow a user to provide a custom template. Without
that, the template selection logic will fallback to using the default template
setup in the global section of the config file, causing the ASG to adopt
unintended scaling values. This change makes it query CloudFormation instead, as
in the case of a cluster update.
Fixes #325.
Signed-off-by: Raghu Raja<EMAIL_ADDRESS>By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Will do some more testing before merging this.

Updated to address offline comments from @mohanasudhan.


Added CircleCI

Fixed some golangci issues
Added Makefile commands

@metanerd - Please check for CircleCI best practices and let me know if there's something you want changed. Thanks!


Maven failed to build this java package

General information

SDK/Library version: JAVA 11
Environment: Sandbox
Language, language version, and OS: Java 11 on Windows 7

Issue description
Maven failed to build this package.  Please see attachment for detail.  Based on the ReadMe, I enter "mvn package" command to build this package with no success.  Any suggestion?
Thanks,

SDK/Library version: JAVA 11

What version of the Braintree SDK are you trying to download? You put the version of Java you are using instead.

@BrianHo888 The issue does not seem to be related to the library itself, but rather to the javac provided by JDK 11. If you will try to compile any class, even one that does not belong to the project using plain javac compiler with 1.5 source/target parameters set you will get error message like this
warning: [options] bootstrap class path not set in conjunction with -source 5
error: Source option 5 is no longer supported. Use 6 or later.
error: Target option 1.5 is no longer supported. Use 1.6 or later.

You can check it on yourself by running command
javac SomeClass.java -source 1.5 -target 1.5

For sure you should be able to compile the library using JDK 8.

SDK/Library version: JAVA 11

What version of the Braintree SDK are you trying to download? You put the version of Java you are using instead.

I downloaded it from: https://github.com/braintree/braintree_java

@BrianHo888 The issue does not seem to be related to the library itself, but rather to the javac provided by JDK 11. If you will try to compile any class, even one that does not belong to the project using plain javac compiler with 1.5 source/target parameters set you will get error message like this
warning: [options] bootstrap class path not set in conjunction with -source 5
error: Source option 5 is no longer supported. Use 6 or later.
error: Target option 1.5 is no longer supported. Use 1.6 or later.

You can check it on yourself by running command
javac SomeClass.java -source 1.5 -target 1.5

For sure you should be able to compile the library using JDK 8. Also suggestion from error message to force minimum version of source/target Java version 1.6 will do the trick.

@BrianHo888 The issue does not seem to be related to the library itself, but rather to the javac provided by JDK 11. If you will try to compile any class, even one that does not belong to the project using plain javac compiler with 1.5 source/target parameters set you will get error message like this
warning: [options] bootstrap class path not set in conjunction with -source 5
error: Source option 5 is no longer supported. Use 6 or later.
error: Target option 1.5 is no longer supported. Use 1.6 or later.

You can check it on yourself by running command
javac SomeClass.java -source 1.5 -target 1.5

For sure you should be able to compile the library using JDK 8. Also suggestion from error message to force minimum version of source/target Java version 1.6 will do the trick.

Thanks for the clarification.  It looks like I will need to fallback to JDK 8 and see if I can compile it.  Oracle will end JDK 8 support in 1/2019.  I upgraded to JDK 11 from https://www.oracle.com/technetwork/java/javase/overview/index.html

This should no longer be an issue as of 2.96.0


add(clientStore): try to speed up things

workflows:

update runs-on

code:

instead of looking up the clientId 2 times with find we bulk with both in same lookup
URL is deprecated, use URI().toURL()
add findClients to clientRegistry
add findClients to ClientStore, to map
update upserQuery to only do update if there is changes to data

mock:

update mocks

Tests

tests new behaviour to storeClients
add test for findClients
simple benchmark, comparing find and findClients

consider:
validating audience input
cache of client data

ikke bry om den detekt, skal fjerne det å bruke jmailen.kotlinter men tar det i en egen PR etter denne


drop support for nodejs 0.12.x

Only get and delete will be allowed for nodejs 0.12.x actions (everything else has been removed).

pg2#1229

anyone understand why pg and travis results are so different?

Sounds like an issue with pg build - open in blue?

pg3#405


In dark mode, the Rust icon is almost unnoticeable

While using dark mode on Github, the Rust icon is unnoticeable, which gives the impression that the exercise is missing the Rust implementation.
Tried to use inline CSS in this PR, but it's not ideal since it wouldn't work well in other versions.
Ideally, we should overcome that by changing directly the SVG, maybe.

I would like to work on this issue to achieve the required changes using dark mode on Github . Please assign it to me.


Backtesting ?

Hello there,
Currently I am wondering how far the back testing has been tested & measured. As for my self I am fiddling around with it and its quite hard to believe the following values.
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) start time:                      2016-06-01 22:49:00
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) end time:                        2016-06-27 23:51:00
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) timespan:                        a month days

2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) start price:                     0.0260104
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) end price:                       0.02145845
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) Buy and Hold profit:             -17.500500000000002%

2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) amount of trades:                823
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) original simulated balance:      0.04000 BTC
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) current simulated balance:       0.39447 BTC
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) simulated profit:                0.35447 BTC (886.18082%)
2016-06-30 08:17:30 (INFO):     (PROFIT REPORT) simulated yearly profit:         4.96802 BTC (12420.04804%)

Exchange:
Poloniex
Alchorhytm: StochRSI
CandleSize: 5
Settings:
// StochRSI settings
config.StochRSI = {
  interval: 3,
  thresholds: {
    low: 28,
    high: 59,
    // How many candle intervals should a trend persist
    // before we consider it real?
    persistence: 3
  }
};

Hey, thanks for reporting backtesting results! I did some testing under different variables and I did not find any extraordinary results.
For those profits, could you please post the configuration of the profitSimulator?

Heya! Your most welcome!
Here are the Profit Simulator settings:

// do you want Gekko to calculate the profit of its own advice?
config.profitSimulator = {
enabled: false,
// report the profit in the currency or the asset?
reportInCurrency: true,
// start balance, on what the current balance is compared with
simulationBalance: {
// these are in the unit types configured in the watcher.
asset: 0,
currency: 0.04,
},
// only want report after a sell? set to false.
verbose: false,
// how much fee in % does each trade cost?
fee: 0.25,
// how much slippage/spread should Gekko assume per trade?
slippage: 0.05
}

Regarding the profit simulator settings, I am actually making loss on it which is a few BTC Each trade.
Started on the Exchange with 0.04 and currently at 0.036 so there is some thing that indicates some thing wrong. As far from what I have checked/seen so far is that its actually a -17% Loss over a month which is indicated on the top at the 'Buy and Hold' profit.

So I want to run the same test as you so I can figure this out, could you:

let me know what market you were backtesting on?
send me the full logs (all 2016-06-30 10:40:11       Sell  0.01876000     2.01008485   0.03770919 messages).
tell me how you obtained the market data (assuming through Gekko's importer?)

Thanks!

Sure thing,
Unfortunately I got tackled by my 10 months old niece, will supply them shortly. The market I am using is BTC/ETH.
Making a fresh import and giving the fresh backtest in ~10 Minutes.

Here we go:
Configuration is exactly the same as submitted earlyer and I have used the internal importer tool.
Settings (from config.js)
// Monitor the live market
config.watch = {
  exchange: 'poloniex', // see gekko/docs/supported_exchanges.md
  currency: 'BTC',
  asset: 'ETH'
}

Trading ADvisor
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
//                       CONFIGURING TRADING ADVICE
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

config.tradingAdvisor = {
  enabled: true,
  method: 'StochRSI',
  candleSize: 6,
  historySize: 25,
  adapter: 'sqlite',
  talib: {
    enabled: true,
    version: '1.0.2'
  }
}

StochRSI Settings
// StochRSI settings
config.StochRSI = {
  interval: 3,
  thresholds: {
    low: 28,
    high: 59,
    // How many candle intervals should a trend persist
    // before we consider it real?
    persistence: 3
  }
};

The output can be found:
https://gist.github.com/RDash21/790e67abc7c000af8bd21ef514e870de

Oh damn, that is a big output (I always run a full backtest, say more than 2 hours, with debug turned off), but that will definitely help me figure this out. Going to look at it!

Debug is turned off and this backtest is based of a Month ¯_(ツ)_/¯

Ah you are right, the stochRSI uses a log level that outputs regardless of debug settings..

vibrates
Although that backtest is valid ? or did I hit a bug? :)

Update:
Apparently it gives still the same out put regardless of debug on/off.
Question:
Is this a bug I have hit or is this information valid?
-Dash

Update:
Put the candle size from 6 to 80 which made allot more sense:
Output:
2016-06-30 15:01:19 (INFO):     Profit simulator got advice to short    @ 2016-06-28 23:22:00, selling 3.05656084 ETH

2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) start time:                      2016-06-01 22:49:00
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) end time:                        2016-06-29 00:01:00
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) timespan:                        a month days

2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) start price:                     0.0260104
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) end price:                       0.01879273
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) Buy and Hold profit:             -27.749170000000007%

2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) amount of trades:                49
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) original simulated balance:      0.04000 BTC
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) current simulated balance:       0.05689 BTC
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) simulated profit:                0.01689 BTC (42.22540%)
2016-06-30 15:01:19 (INFO):     (PROFIT REPORT) simulated yearly profit:         0.22791 BTC (569.76972%)

Hey @RDash21,
I finally had some time to look at all the moving pieces:

Apparently it gives still the same out put regardless of debug on/off.

This was a bug in the trading method and is fixed now. I overall refactored and fixed some bugs in the stochRSI code.

its quite hard to believe the following values.

I fixed a major bug in the backtesting code (I need more unit tests!), running a very similar backtest  (using a candleSize of 6, from 2016-06-01 22:49:00 to 2016-06-25 23:51:00) now yields:
2016-06-30 22:12:05 (INFO): (PROFIT REPORT) amount of trades:    390
2016-06-30 22:12:05 (INFO): (PROFIT REPORT) original simulated balance:  100.02601 BTC
2016-06-30 22:12:05 (INFO): (PROFIT REPORT) current simulated balance:   24.78887 BTC
2016-06-30 22:12:05 (INFO): (PROFIT REPORT) simulated profit:    -75.23714 BTC (-75.21758%)
2016-06-30 22:12:05 (INFO): (PROFIT REPORT) simulated yearly profit:   -1142.14953 BTC (-1141.85253%)

Could you please update gekko, test again and let me know the results?

2016-07-01 03:54:22 (INFO):     (PROFIT REPORT) amount of trades:                897
2016-07-01 03:54:22 (INFO):     (PROFIT REPORT) original simulated balance:      0.04000 BTC
2016-07-01 03:54:22 (INFO):     (PROFIT REPORT) current simulated balance:       0.00887 BTC
2016-07-01 03:54:22 (INFO):     (PROFIT REPORT) simulated profit:                -0.03113 BTC (-77.82191%)
2016-07-01 03:54:22 (INFO):     (PROFIT REPORT) simulated yearly profit:         -0.42080 BTC (-1052.00992%)

Good morning,
Yes! It is fixed now! :+1:


Oculus Quest freezes when steamvr is open

I am running 2.4.0-alpha5, i can connect my headset with my pc no problem but as soon as i launch steamVr the headset freezes on a whitescreen and nothing happens. If i go to steamvr on my computer and view the VR Viewer i can see from the headsets perspective. Its tracking completly fine, i can even see the controllers too but it just wont display on the headset.

For Oculus Quest you can try JackD83's fork.
It works only with Quest, and it has several Quest specific improvements.


Tests JSON schema v4 upgrade

Linked to @jmarrec's PR to upgrade json_schema to v4. Previous v2.7 (i.e. any version < 3) had been maintained to ensure compatibility with OpenStudio v2.9.1 ... which we no longer support. All tests are green locally.

Thanks for looking into this, @macumber ! Can now reproduce the issue locally (Ruby 3.2.2. + OS v.3.8.0):
:bundle install
Fetching https://github.com/rd2/tbd
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Could not find compatible versions

Because every version of tbd_tests depends on Ruby ~> 2.7.0
  and Gemfile depends on tbd_tests >= 0,
  Ruby ~> 2.7.0 is required.
So, because current Ruby version is = 3.2.2,
  version solving has failed.


Make more constructors return &PyX instead of Py

fixes #405
Remaining one is PyTuple, where I'm running into a test failure. Will provide as a separate PR.

Missing changelog entry; I'd like to combine this with the one for PySet when both are merged.

CC @pganssle for the datetime constructors

The datetime constructors broke a bunch of tests. I have a half-completed branch somewhere that fixes the tests, but IIRC it was semi-blocked on #392.
@birkenfeld If you don't want to be bothered with fixing the datetime changes, I can handle them later.

@pganssle I've found them now :smile:
Do you know how to change the utcoffset method? Trying to make it return PyResult<&'p PyDelta> makes the proc-macro complain:
190 | #[pymethods]
    | ^^^^^^^^^^^^
    |
    = help: message: python method can not be generic: Ident { ident: "utcoffset", span: #0 bytes(4224..4233) }

The pyfunctions seem to be fine with it. Although, trying to import that module makes python3 segfault...

Do you know how to change the utcoffset method? Trying to make it return PyResult<&'p PyDelta> makes the proc-macro complain

Changing it to return Py<PyDelta> works but I think it's better to separate datetime fixes into an other PR.

Sorry, can you be more specific what you want to separate? Without the datetime changes there isn't much left here...

Also it's unclear to me how to get the Py from the reference :smile:

Without the datetime changes there isn't much left here...

Yeah but there's many examples for datetime module and I thought it's tough stuff to fix all of them.
But spending after some time fixing this, I found it not too much difficult.

Also it's unclear to me how to get the Py from the reference

use pyo3::AsPyPointer;
#[pyfunction]
fn make_date(py: Python<'_>, year: i32, month: u8, day: u8) -> PyResult<Py<PyDate>> {
    PyDate::new(py, year, month, day).map(|d| unsafe { Py::from_borrowed_ptr(d.as_ptr()) })
}

It works but maybe we need an easier way.

use pyo3::AsPyPointer;
#[pyfunction]
fn make_date(py: Python<'_>, year: i32, month: u8, day: u8) -> PyResult<Py<PyDate>> {
    PyDate::new(py, year, month, day).map(|d| unsafe { Py::from_borrowed_ptr(d.as_ptr()) })
}

This works but maybe we need an easier way.

Uh oh. Should unsafe really be required here?

I think we need a generic impl of that snippet:
https://github.com/PyO3/pyo3/blob/874d8a0835ff0d0d8239108d01bb2d1628918e42/src/types/tuple.rs#L137-L141
This would also avoid unsafe in downstream code

This seems to work:
impl<'a, T: AsPyPointer> FromPy<&'a T> for Py<T> {
    fn from_py(tuple: &'a T, _py: Python) -> Py<T> {
        unsafe { Py::from_borrowed_ptr(tuple.as_ptr()) }
    }
}

Might be good idea to investigate this for soundness and make another pull request of that.

@konstin
Yeah I agree.
I noticed we can return PyResult<&T> for some pyfunctions in the datetime example.
E.g.,
#[pyfunction]
fn make_date(py: Python, year: i32, month: u8, day: u8) -> PyResult<&PyDate> {
    PyDate::new(py, year, month, day)
}

But when we have &PyTzInfo as an argument, it doesn't work for the lifetime reason. We have to return Py<T> in datetime_from_timestamp and make_datetime.
#[pyfunction]
fn datetime_from_timestamp(py: Python, ts: f64, tz: Option<&PyTzInfo>) -> PyResult<&PyDateTime> {
    PyDateTime::from_timestamp(py, ts, tz)
}

I think the latter part can be resolved by proc-macro hack, but not sure.

@kngwyu Why can't we use:
fn datetime_from_timestamp(py: Python<'p>, ts: f64, tz: Option<&'p PyTzInfo>) -> PyResult<&'p PyDateTime> {

The PyTzInfo just has to live as long as the PyDateTime or longer, so the annotation seems correct.

@pganssle
We can't use the lifetime specifier py there. It's the limitation of our proc macro codes.
So the compiler can't know python and tz has the same lifetime.

@pganssle
Hoops, sorry I noticed this works now.
#[pyfunction]
fn datetime_from_timestamp<'py>(
    py: Python<'py>,
    ts: f64,
    tz: Option<&'py PyTzInfo>,
) -> PyResult<&'py PyDateTime> {
    PyDateTime::from_timestamp(py, ts, tz)
}

But now methods can't take lifetime specifiers so we can't compile
    fn utcoffset<'py>(&self, py: Python<'py>, _dt: &'py PyDateTime) -> PyResult<&'py PyDelta> {
        PyDelta::new(py, 0, 3600, 0, true)
    }

Whoops, rustfmt fixed.

Thank you!


Rc/0.11.2 sp0

Sorry, I meant to open this against our local fork. Once I see that everything upgrades properly, I'll submit a PR here if there's interest in using InBBox on multiple attributes 😄


[SPARK-34093][ML] param maxDepth should check upper bound

What changes were proposed in this pull request?
update the ParamValidators of maxDepth
Why are the changes needed?
current impl of tree models only support maxDepth<=30
Does this PR introduce any user-facing change?
If maxDepth>30, fail quickly
How was this patch tested?
existing testsuites

scala> import org.apache.spark.ml.classification._
import org.apache.spark.ml.classification._

scala>

scala> val df = spark.read.format("libsvm").load("/d0/Dev/Opensource/spark/data/mllib/sample_multiclass_classification_data.txt")
21/01/13 11:40:35 WARN LibSVMFileFormat: 'numFeatures' option not specified, determining the number of features by going though the input. If you know the number in advance, please specify it via 'numFeatures' option to avoid the extra scan.
df: org.apache.spark.sql.DataFrame = [label: double, features: vector]

scala> val dt = new DecisionTreeClassifi
DecisionTreeClassificationModel   DecisionTreeClassifier

scala> val dt = new DecisionTreeClassifier().setMax
setMaxBins   setMaxDepth   setMaxMemoryInMB

scala> val dt = new DecisionTreeClassifier().setMaxDepth(32)
dt: org.apache.spark.ml.classification.DecisionTreeClassifier = dtc_9ad3d8b78d64

scala> dt.fit(df)
21/01/13 11:41:01 ERROR Instrumentation: java.lang.IllegalArgumentException: requirement failed: DecisionTree currently only supports maxDepth <= 30, but was given maxDepth = 32.
	at scala.Predef$.require(Predef.scala:281)
	at org.apache.spark.ml.tree.impl.RandomForest$.runBagged(RandomForest.scala:145)
	at org.apache.spark.ml.tree.impl.RandomForest$.run(RandomForest.scala:302)
	at org.apache.spark.ml.classification.DecisionTreeClassifier.$anonfun$train$1(DecisionTreeClassifier.scala:135)
	at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
	at org.apache.spark.ml.classification.DecisionTreeClassifier.train(DecisionTreeClassifier.scala:114)
	at org.apache.spark.ml.classification.DecisionTreeClassifier.train(DecisionTreeClassifier.scala:46)
	at org.apache.spark.ml.Predictor.fit(Predictor.scala:150)
	at $line21.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:31)
	at $line21.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:35)

scala> import org.apache.spark.ml.classification._
import org.apache.spark.ml.classification._

scala>

scala> val df = spark.read.format("libsvm").load("/d0/Dev/Opensource/spark/data/mllib/sample_multiclass_classification_data.txt")
21/01/13 11:40:35 WARN LibSVMFileFormat: 'numFeatures' option not specified, determining the number of features by going though the input. If you know the number in advance, please specify it via 'numFeatures' option to avoid the extra scan.
df: org.apache.spark.sql.DataFrame = [label: double, features: vector]

scala> val dt = new DecisionTreeClassifi
DecisionTreeClassificationModel   DecisionTreeClassifier

scala> val dt = new DecisionTreeClassifier().setMax
setMaxBins   setMaxDepth   setMaxMemoryInMB

scala> val dt = new DecisionTreeClassifier().setMaxDepth(32)
dt: org.apache.spark.ml.classification.DecisionTreeClassifier = dtc_9ad3d8b78d64

scala> dt.fit(df)
21/01/13 11:41:01 ERROR Instrumentation: java.lang.IllegalArgumentException: requirement failed: DecisionTree currently only supports maxDepth <= 30, but was given maxDepth = 32.
	at scala.Predef$.require(Predef.scala:281)
	at org.apache.spark.ml.tree.impl.RandomForest$.runBagged(RandomForest.scala:145)
	at org.apache.spark.ml.tree.impl.RandomForest$.run(RandomForest.scala:302)
	at org.apache.spark.ml.classification.DecisionTreeClassifier.$anonfun$train$1(DecisionTreeClassifier.scala:135)
	at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
	at org.apache.spark.ml.classification.DecisionTreeClassifier.train(DecisionTreeClassifier.scala:114)
	at org.apache.spark.ml.classification.DecisionTreeClassifier.train(DecisionTreeClassifier.scala:46)
	at org.apache.spark.ml.Predictor.fit(Predictor.scala:150)
	at $line21.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:31)
	at $line21.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:35)

Test build #133995 has started for PR 31163 at commit 1d699e5.

Test build #133995 has started for PR 31163 at commit 1d699e5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38583/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38583/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38583/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38583/

Test build #133995 has finished for PR 31163 at commit 1d699e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133995 has finished for PR 31163 at commit 1d699e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133995/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38583/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133995/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38583/

Test build #134174 has started for PR 31163 at commit 9cce6ff.

Test build #134174 has started for PR 31163 at commit 9cce6ff.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38759/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38759/

Test build #134174 has finished for PR 31163 at commit 9cce6ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134174 has finished for PR 31163 at commit 9cce6ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134174/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134174/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38759/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38759/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38759/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38759/

Merged to master

Merged to master


ARROW-15979: [C++][Doc]Expose more functions of parquet::WriterProperties in doc

Ah, I missed that. Yes, it just sets some metadata about what implementation produced the Parquet file. We can document it, but probably people won't want to/shouldn't change it.

Guess it is able to merge? The default row group size can be fixed in another PR. created_by does not need to expose in doc.


append-snippets doesn't work on Contexts other than FeatureContext

When I use the --dry-run --append-snippets option it tells me it worked, but it doesn't actually append the snippets to a context other than FeatureContext (and it didn't put them in there either). This worked before I set up the behat.yml to specify another Context.
behat.yml:
default:
    suites:
        user:
            paths:    [ %paths.base%/features/user ]
            contexts: [ UserRegistrationContext ]

I create a new feature and run behat, and get:
> vendor\bin\behat
Feature: New User Signs Up
  As a potential user
  I need to be able to register
  So that I can play the game

  Scenario: Successful Registration                                                 # features\user\new-user.feature:6
    Given I am registering for the game
    When I provide an unused valid username, a password and an unused email address
    And I submit my registration
    Then I should be registered
    And I should see a success message telling me to confirm my email address

1 scenario (1 undefined)
5 steps (5 undefined)
0m0.04s (6.22Mb)

--- UserRegistrationContext has missing steps. Define them with these snippets:

    /**
     * @Given I am registering for the game
     */
    public function iAmRegisteringForTheGame()
    {
        throw new PendingException();
    }

    /**
     * @When I provide an unused valid username, a password and an unused email address
     */
    public function iProvideAnUnusedValidUsernameAPasswordAndAnUnusedEmailAddress()
    {
        throw new PendingException();
    }

    /**
     * @Given I submit my registration
     */
    public function iSubmitMyRegistration()
    {
        throw new PendingException();
    }

    /**
     * @Then I should be registered
     */
    public function iShouldBeRegistered()
    {
        throw new PendingException();
    }

    /**
     * @Given I should see a success message telling me to confirm my email address
     */
    public function iShouldSeeASuccessMessageTellingMeToConfirmMyEmailAddress()
    {
        throw new PendingException();
    }

When I run with --dry-run --append-snippets, it tells me it appended them, but the file does not get updated, and the next attempt to run behat gives me the same --- UserRegistrationContext has missing steps. Define them with these snippets:
> vendor\bin\behat --dry-run --append-snippets
Feature: New User Signs Up
  As a potential user
  I need to be able to register
  So that I can play the game

  Scenario: Successful Registration                                                 # features\user\new-user.feature:6
    Given I am registering for the game
    When I provide an unused valid username, a password and an unused email address
    And I submit my registration
    Then I should be registered
    And I should see a success message telling me to confirm my email address

1 scenario (1 undefined)
5 steps (5 undefined)
0m0.03s (6.16Mb)

u features\bootstrap\UserRegistrationContext.php - `I am registering for the game` definition added
u features\bootstrap\UserRegistrationContext.php - `I provide an unused valid username, a password and an unused email address` defini
tion added
u features\bootstrap\UserRegistrationContext.php - `I submit my registration` definition added
u features\bootstrap\UserRegistrationContext.php - `I should be registered` definition added
u features\bootstrap\UserRegistrationContext.php - `I should see a success message telling me to confirm my email address` definition
added

Removing the             contexts: [ UserRegistrationContext ] from my behat.yml and running with --append-snippets puts the snippets in the FeatureContext file.

Can not replicate on latest version. So I assume it was either fixed somewhere down the road or it is something very weird and environmental for you.

Please feel free to reopen if you can replicate on latest 3.1 version.

Thanks @everzet - I'll update to 3.1 and try again. I am on Windows, I should have mentioned that, so it could be environmental, a lot of times people forget to check if file writing things work on Windows and often they don't, cause... Windows ;)

I can confirm this works with behat v3.1
# behat.yml
default:
  autoload:
    '': %paths.base%/features/contexts
  suites:
    general:
      paths: [ %paths.base%/features/general ]
      contexts: [ MyGeneralContext ]
    order_requests:
      paths: [ %paths.base%/features/other ]
      contexts: [ MyOtherContext ]

@everzet  I was able to reproduce this behavior. I'm on Windows OS. I have been able to understand the problem. It seems that when the custom context class implements Context the --dry-run --append-snippets doesn't generate the snippets. However if the context in my case extends the MinkContext everything work fine.

@Hermannovich this is a different problem and 3.2.1 (just released) should fix it for you.

Thanks.


GCE create_node() with location=None (default) throws AttributeError

Summary
Running with the GCE driver using create_node(.. , location=None) (which is a default) throws an AttributeError
Detailed Information
OS: Ubuntu 19.10
libcloud Version: apache-libcloud==2.8.1
Python version: Python 3.7.5
Code to reproduce:
gce = get_driver(Provider.GCE)(
<EMAIL_ADDRESS>            api_key_file,
            project=project_id)

gce.create_node(name, size, None,
                             ex_network=network,
                             ex_subnetwork=subnet,
                             ex_tags=tags,
                             ex_metadata=metadata,
                             external_ip='ephemeral',
                             internal_ip=internal_ip,
                             ex_image_family=image,
                             ex_disk_size=disk_sz)

With exception:
Traceback (most recent call last):
[ .. snip .. ]
  File "/home/kevin/.local/lib/python3.7/site-packages/proj/gcp/builder.py", line 181, in create_vm
    ex_disk_size=disk_sz)
  File "/home/kevin/.local/lib/python3.7/site-packages/libcloud/compute/drivers/gce.py", line 4169, in create_node
    location = self.ex_get_zone(location)
  File "/home/kevin/.local/lib/python3.7/site-packages/libcloud/compute/drivers/gce.py", line 7811, in ex_get_zone
    if name.startswith('https://'):
AttributeError: 'NoneType' object has no attribute 'startswith'

Thanks for reporting this.
It's likely related to recent **kwargs refactoring. I will have a look.

I had a look and the issue is related to not passing datacenter argument to the driver constructor.
Location needs to be specified one way or another - either via constructor datacenter argument or via location argument to the create_node method.

Having said that - I will also open a PR so an exception is throw if neither of those is provided and clarify that in the docs.
Thanks again for reporting this.


score Celeste's half-life radius on stripe 82 correctly

@gostevehoward identified a bug in how galaxy scales are compared while working with galsim data (issue #437). The same problem affects scoring on stripe 82. With the change, scores from validate_on_stripe82.jl look very good!
│ Row │ field        │ primary   │ celeste   │ diff       │ diff_sd   │ N   │
├─────┼──────────────┼───────────┼───────────┼────────────┼───────────┼─────┤
│ 1   │ position     │ 0.372055  │ 0.238824  │ 0.133232   │ 0.0658357 │ 421 │
│ 2   │ missed_stars │ 0.023753  │ 0.0902613 │ -0.0665083 │ 0.013295  │ 421 │
│ 3   │ missed_gals  │ 0.0570071 │ 0.0190024 │ 0.0380048  │ 0.010859  │ 421 │
│ 4   │ mag_r        │ 0.20105   │ 0.362753  │ -0.161702  │ 0.021012  │ 421 │
│ 5   │ color_ug     │ 1.25429   │ 0.697098  │ 0.557196   │ 0.0549773 │ 374 │
│ 6   │ color_gr     │ 0.3749    │ 0.214061  │ 0.160839   │ 0.0487459 │ 418 │
│ 7   │ color_ri     │ 0.253103  │ 0.169953  │ 0.0831497  │ 0.0493487 │ 421 │
│ 8   │ color_iz     │ 0.308664  │ 0.150345  │ 0.158319   │ 0.0197942 │ 420 │
│ 9   │ gal_fracdev  │ 0.258861  │ 0.314312  │ -0.0554506 │ 0.0237844 │ 175 │
│ 10  │ gal_ab       │ 0.190628  │ 0.126675  │ 0.0639529  │ 0.0105222 │ 175 │
│ 11  │ gal_scale    │ 1.63777   │ 1.75603   │ -0.118254  │ 0.622799  │ 175 │
│ 12  │ gal_angle    │ 17.0412   │ 12.6401   │ 4.40112    │ 1.79078   │ 80  │

Now Celeste is better than Photo/primary, or statistically tied (2 SDs), on every metric except missed gals and mag_r (brightness). The former problem may be patched up in issue #390. The latter problem is very likely due to inaccuracies in what we're treating as the ground truth, that systematically favor Photo.

Some recent changes to Julia 0.6 seem to have broken some of our unit tests. This PR fixes those too.

@kpamnany @andreasnoack @jrevels @keno
This PR also adds a new benchmark named benchmark_quarter_degree.jl. It's the best benchmark to use if you're testing Celeste with multiple threads. It optimizes all the light sources (> 2000) in a region of sky that is one quarter square degree.  That's about the area as we anticipate processing for each node-level task. With 4 threads, it currently takes about 10 minutes to complete on Cori phase 1.


Is the initPayload not triggered in the javascript version?

Hey,
I've read the other questions about posting a "welcome message" at the start of the chat and followed all the steps mentioned in those.
I am using the script-tag variance, not the React component. Does the script-tag one not support the initPayload?
I added the following to my domain.yml:
intents:
- get_started

templates:
  utter_introduction1:
  - text: Hello, how can I help you?

actions:
- utter_introduction1

Then this to my stories:
## welcome
* get_started
    - utter_introduction1

(as well as several other stories, which start with *get_started and the utterance)
and my script tag includes the
initPayload: "/get_started",

As far as storage goes, the script tag is set to
storage: "session"

I also tried adding a /get_started to my NLU under the intent:get_started, but since it was never mentioned anywhere to do this, I removed it and tried that as well.
Whenever I made a change, I executed a rasa train and a new model was created.
Before calling the WebChat.default.init, I also tried to put a sessionStorage.clear(), but it also does nothing.
The chatbot itself works fine, but it's just reactive, it doesn't start the conversation; the user has to do that, which isn't great.
Is there anything I have to consider in the socketio nodejs file? Like handling the initPayload or something like that? Because I can't find information about that anywhere and it's never mentioned when other people have this problem.
I'd really appreciate some help, thank you!

It all seems correct. The only thing I can think of is that there's already something stored in your session and for that reason it doesn't send the initPayload. Can you try playing with the various storage options, and also see if changing the session_persistence param in the Rasa credentials help?

I'll try that on Monday, thank you very much!

I set initPayload to a text value that a user would normally have typed. E.g., initPayload: "hello". This then maps back to my greet intent and greets the user on first interaction. It's working fine for me this way using both script and React types. Just make sure your bot response already to whatever phrase you're using in that parameter, and as mentioned above, reset your page cache/storage to check too.

I have the same issue, the initPayload is not always sent.
@psi43: did you find a solution?


ci: better docker curl healthcheck

Codecov Report

Merging #13 (2912183) into master (eafc90c) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master       #13   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           10        10
  Lines          159       159
=========================================
  Hits           159       159

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eafc90c...2912183. Read the comment docs.


[WIP] Add a simple storage class constraint

@morgante is this sill WIP?

I don't think this constraint is practically very useful, so closing it.


Improve logging so that log content can be captured by datadog

Improve logging such that format is compatible with Datadog. Currently no log content is showing in the logs we are getting from Cloudwatch, as such we only get small excerpts of the actually log activity, as well if we want to use any values logged as dd datasources for alerts or charts they must show in log content.

adding to this, the log groups themselves should be distinctly named across availability zones if possible ex /eu-west-1/ecs/gateway


Feature/service user manager

Fixes #892
More i think about it, the more I like OSGi based definitions vs in-code Annotation based. Open to debate; or potentially could have both approaches (if the latter is contrib'd later)

@andrewmkhoury @justinedelson @bstopp  - WDYT?

thoughts on how to achieve the blacklist? Manually curate it?

I guess manually curate it.  You could use the json dump feature of JCR compare to get all the user ids from /home/users/system.  Then grep for a list of rep:principalName and generate the code for a HashSet.

On Feb 17, 2017, at 5:05 AM, david g<EMAIL_ADDRESS>wrote:
thoughts on how to achieve the blacklist? Manually curate it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I thought there was a unique index on rep:principalName, so you can't overwrite an existing name.

They are unique, you can't have two users with the same rep:principalName.
However, I'm more concerned about the ACLs.  To avoid wiping out the ACLs
we would need to check against a list of the OOTB users.
On Fri, Feb 17, 2017 at 8:37 AM, justinedelson<EMAIL_ADDRESS>wrote:

I thought there was a unique index on rep:principalName, so you can't
overwrite an existing name.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/906#issuecomment-280699803,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACQ1gvDdX4-z7BXDBO_j9rTLcD9vqKfmks5rdcy4gaJpZM4L-kv2
.

oh. got it. nevermind then.

@andrewmkhoury added a protected system users principal name list (based on the list you PM'd me).

Any last thoughts before merging?

Nope, seems good to me.
On Thu, Feb 23, 2017 at 4:30 PM, david g<EMAIL_ADDRESS>wrote:

Any last thoughts before merging?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Adobe-Consulting-Services/acs-aem-commons/pull/906#issuecomment-282166868,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACQ1gh1ieacssop7P3kz12lQFkvybvBvks5rfiSwgaJpZM4L-kv2
.


Add support for loading 3rd party widgets from CDN

I don't see any approve button on this PR, but it looks good to me. Please make sure to include a good description and possibly screenshots in the official PR to nteract.


FR: migrate existing table with fresh table

With current implementation (create_hypertable with migrate_data => true), if table is large could be locked for long time and could also cause "deadlock" for certain condition, as the below quote from doc suggest:

WARNING:The use of the migrate_data argument to convert a non-empty table can lock the table for a significant amount of time, depending on how much data is in the table. It can also run into deadlock if foreign key constraints exist to other tables.
If you would like finer control over index formation and other aspects of your hypertable, follow these migration instructions instead.
When converting a normal SQL table to a hypertable, pay attention to how you handle constraints. A hypertable can contain foreign keys to normal SQL table columns, but the reverse is not allowed. UNIQUE and PRIMARY constraints must include the partitioning key.
The deadlock is likely to happen when concurrent transactions simultaneously try to insert data into tables that are referenced in the foreign key constraints and into the converting table itself. The deadlock can be prevented by manually obtaining SHARE ROW EXCLUSIVE lock on the referenced tables before calling create_hypertable in the same transaction, see PostgreSQL documentation for the syntax.

Related docs - https://docs.timescale.com/latest/getting-started/migrating-data#same-db


Re-introduce mandatory name for switch/flag

To make this a less intrusive change we could also introduce case class NameAndMeta(name: Name, meta: Meta) where short/long/both return that with the empty meta.
EDIT: Actually that still doesn't (necessarily) help, or at least it would have a monoid so the call site will still have to change.

:+1:


readiness error

17:45:00 [Error] Readinesshttps://**************/artifactory/api/v1/system/readiness failed
exit status 1
The benchmark api url doesn't exists on Artifactory 7.10 enterprise

That's right! @tamirdavid , we need to add a condition to use the old artifactory/api/system/ping for older versions. I don't remember the exact version. It was 7.2x.x IIRC.

Thank you for informing me. The issue has been resolved after testing on a local Artifactory server with an older version.
$ ./benchmark dl --size 30 --iterations 4 --url http://xxxx --username xx --password bbb
11:19:33 [🔵Info] Validate url is an Artifactory server by sending readiness request [http://xxxx/artifactory/api/v1/system/readiness]
11:19:33 [🟠Warn] Validation through the readiness check has failed, potentially due to the use of an old Artifactory version. Attempt to validate using the system/ping.
11:19:33 [🔵Info] Validate url is an Artifactory server by sending system/ping request [http://xxxx/artifactory/api/system/ping]


Feature x

Description
This PR intends to build new functionality

If true safeshutdown will be triggered; if false completion will be awaited and system will be prepared for shutdown with the help of PrepareForShutdown api, this will not only monitor in_progress asynchronous jobs and wait for its completion and prevent addition of new asynchronous jobs.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

Hi @Yashikaone thanks for your PR, I can see your changes on the second commit, however the first commit is not needed since there are no changes on it, can you please amend the PR to reflect only your changes?

@Yashikaone , you should completely remove commit ce70939f3a3aeb9d0c7f0dda10aa5e531c461691 from your PR.
As the current status is you can do that by entering the command
git rebase -i HEAD~5 and then in the editor you get presented remove the line for that commit from the file. Git will remnove the commit. After the complete procedure check the result. You can then git push -- force and the commit with all the empty changes will be gone.

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6662 (SL-JID-2231)

This was superseeded by https://github.com/apache/cloudstack/pull/6755 closing on this remark. Thanks for the PR.


Add proper documentation

We could use the wiki feature of Github, or add a 'docs' folder with markup for each module.
If we wanted to be really clever, maybe we could parse routine comments? Maybe @diegoboston 's area of expertise?

Do you want to implement *csdoc ? :-)
It's probably not too hard if you use the right tools, but time consuming.

Do you want to implement *csdoc ? :-)
It's probably not too hard if you use the right tools, but time consuming.

It would be nice, if it saves time over writing them manually. But, you're right, it probably won't.

how would you implement it, hypothetically speaking, @diegoboston? 🥺


[WIP] Run the Calico UTs inside a container

Turns out I'd already written this as a learning exercise. Still has some kinks to work out. Example invocation:
$ cd /path/to/calico/code
$ docker build -t calico/ut .
$ docker run -v $(pwd):/calico calico/ut

Still to do:

[ ] Fix the bug that prevents you using tox on the host and in a container
[ ] Use run-unit-tests.sh instead of plain tox
[ ] Work out where to put this in the repo (any suggestions?)
[ ] Update README
[ ] Add a Makefile

One other thing that springs to mind – are we happy that UTs passing on one Linux flavour is sufficient, or do we want a RHEL variant as well?

@fasaxc Turns out it was fairly easy to tidy this up. Could you review please?
My only concern is that it's a big image. (630MB) Unfortunately, I don't see an easy way to slim down the layers: https://imagelayers.io/?images=alexwlchan%2Fcalico- Will that be a problem?

Actually, it occurs to me that I should rebase with your changes for tox coverage before review.

I’ve rebased to include Shaun’s changes to tox/coverage. I’d test it, but this container doesn’t work on OS X because distutils tries to create a hard link when it installs the package, and VirtualBox/Docker really don’t like that. That means these UTs will only work on Linux, boo.
It is possible to do some distutils jiggery-pokery to make this work, but I want to think carefully before I actually do it.

I think this is a good start but I'm wondering if we need to join it up with a few other areas before we merge it:

@tomdee has #849  to add a Dockerfile for Felix.  This adds another one with different setup for the UTs , which feels like a lot of duplication :-(
How does this relate to the run-unti-tests.sh script that is used by Jenkins?  Should we make containers the official way to run the UTs and integrate this all together?
PyCharm 5.0 has container support (that's all I know) but I'm not sure if it works with this approach.  Feels like we should close that loop.
Once we have containers, do we still want tox?  Would it be better to run separate containers with the different levels of Python and do real installs instead of hokey virtualenv installs?

Apart from that, I think you need to address cleanup of unused container images and containers.  They quickly stack up and fill up the disk IME.
To get the size of the container down, you could use one of these ("-slim") images: https://hub.docker.com/_/python/.

Also: the UTs run as root inside the container, which means any files you map in or copy out will have bad permissions (my .tox dir ended up owned by root after the first run)

tomdee has #849 to add a Dockerfile for Felix. This adds another one with different setup for the UTs , which feels like a lot of duplication

Yeah, this was mostly written a while back and in isolation. We should discuss if/how we want to tie this stuff together.

How does this relate to the run-unti-tests.sh script that is used by Jenkins? Should we make containers the official way to run the UTs and integrate this all together?

It calls that script, but that's about it. I would prefer making this the one and only way to run UTs, but it needs a bit of work first. Having multiple approaches feels like asking for trouble.

Once we have containers, do we still want tox? Would it be better to run separate containers with the different levels of Python and do real installs instead of hokey virtualenv installs?

tox has quite nice result output (IMO). Could we replicate that with containers?

Apart from that, I think you need to address cleanup of unused container images and containers. They quickly stack up and fill up the disk IME.

Yep. The initial version weighs in at >700 MB, which seems excessive. I also haven't thought about how to get coverage information out.
I don't have any more time to work on this today. I'm going to push another commit which replaces most of the RUN stanzas with a shell script and tries to cut down the image size.

The current image is still ~750MB. That's just too big. :-/

via @fasaxc, this seems like it would be worth looking into: https://github.com/openstack/dox

I’m closing this PR because it’s languishing, I’m unlikely to work on it any further, and at this point, anybody wanting to do this should probably be starting from scratch.


RuntimeError: 'OBDErrorCode' object has no attribute 'formate'

使用obd cluster start cluster-name启动时报错

查看是.obd/plugins/oceanbase-ce/3.1.0目录下的start_check.py文件中一处拼写错误

Thank for feedback. This problem has been fixed in #96


align semantics of insertAll() and friends with insertMultiple() and friends in ORM

As of H6, ORM has operations like insertMultiple() accepting List<?> which implicitly set the batch size to the size of the given list. Reactive has very similar insertAll() operations, but they do not set the batch size, though you can accomplish the same thing by calling reactiveInsertAll(array.length, array).
I think the behavior in ORM is more convenient.
On a similar note: after much consideration, I decided that the operations in ORM should accept List instead of varargs, and that Multiple was less ambiguous than All. I'm wondering if it's worth adding insertMultiple(List) and friends to the reactive API (we don't need to drop or rename the older operations).

Merged, thanks

Cheers


DIFFERENT VIEW FOR EACH ALBUM

Add option to select view for each folder/album. Sometimes i need my specific album sorted by name without sorting everything in app by name. If this already there please tell me.

Duplicate:

#207


Added drop_columns feature; usefull for optimizing cutoffs

If an HDF5 file with larger radial cutoffs and uniform knot spacing exists, it is possible to fit a model with smaller cutoffs without regenerating the HDF5 file. This feature can be used for optimizing the radial cutoffs.
This pull request implements the above feature.
regression/optimize.py provides some useful functions for getting the appropriate columns. These columns correspond to the columns of the HDF5 file (created using larger cutoffs), which should be ignored while fitting the model to lower cutoffs.
'least_squares.WeightedLinearModel.fit_from_file' and 'least_squares.WeightedLinearModel.batched_predict' accepts new argument 'drop_columns'. Providing a list of columns to this argument results in ignoring those columns while fitting and predicting steps.
Appropriate unit tests were added to 'tests/test_optimize.py'

[x] Create an example for the W system demonstrating this feature

I have a checklist in my first comment that talks about adding an example.
Regarding the comment about the code not working when rmin!=0, the code might work when rmin!=0, I just haven't tested it. We should probably keep the condition for now and later remove it when we have extensively tested it.
I will make the changes in line 365 and 490.
I can make get_bspline_config a bit more flexible like accepting different knot spacing for 2-body and 3-body term. Regarding the leading trim and trailing trim, as I said for rmin I haven't had the time to test other settings.
I will add extra checks in get_columns_to_drop_2b and get_columns_to_drop_3b, to check for trims, knot_spacing, and some other stuff.
@JasonGibsonUfl I like your suggestion for renaming the functions.
Suggestions for line 158 and 154 noted.
The ValueError for internal checks is redundant check. For the drop_columns feature to work the smaller cutoff needs to be a knot of the original knots with larger cutoffs.

As per @JasonGibsonUfl suggestions, I have made get_bspline_config a bit more flexible. It now works when rmin!=0.
I added some extra checks to get_columns_to_drop_2b and get_columns_to_drop_3b functions.
Also added one more unit test (see config_2 in test_optimize.py), which is failing with my current get_columns_to_drop_3b algorithm.
The algorithm only fails when the modify_3b_cutoff is either 4.5 or 6.3. It works for other values like for example 5.4, 7.2, 1.8, etc.
@JasonGibsonUfl @sxie22 any suggestions?

The algorithm is only failing when rmin_3b!=0. When rmin_3b=0, it passes all the cases I was able to come up with.

I think I have found the reason why my drop_column algorithm is failing. Refer to issue #89 for more details.

Formated the files according to @JasonGibsonUfl suggestions and also added an example jupyter notebook

Merging!


leftwm: 0.4.1 -> 0.4.2

Description of changes
https://github.com/leftwm/leftwm/releases/tag/0.4.2
Unbreak on aarch64-linux since it builds and runs there.

Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 238626 run on x86_64-linux 1

1 package built:

leftwm


Add noise to streamlines

First real PR!!
function add_noise_to_streamlines.

Hello @EmmaRenauld! Thanks for updating this PR.

In the file dwi_ml/data/processing/streamlines/data_augmentation.py:

Line 39:1: W293 blank line contains whitespace

If I test the formula:
n=truncnorm.rvs(-2, 2, size=(1000,1)), data goes from -2 to 2.
n=truncnorm.rvs(-2, 2, size=(1000,1), scale=4), data goes from -8 to 8.
So I guess that with the scale option, you already kind of clip to +- 2*noise

I have not tested it myself, and I was confused by the scale parameter: the a,b parameters and the scale influence the support of the distribution. So it looks like an alternative way to influence the support. When I first read scale, I thought it meant the amplitude of the distribution, which can be changed otherwise multiplying the vector by the desired factor.
If we want to add Gaussian noise, may be we would want to wrap the method as in here to just provide the mu and sigma values, and may be the clipping factors, as parameters.

Ok, we had a discussion, here is a quick comment to clarify everything:
The goal of adding noise is to jitter the streamline coordinates to make the model robust to small changes in directions, and not be biased to the specific directions of the dataset (especially when using the result of probabilistic tracking). Thus, in average, you should learn a smooth direction.
We also want to use zero-mean noise, because it would not make sense to add positive or negative noise (in euclidean space) to streamline coordinates.
Furthermore, my first comment didn't make sense, the noise is already clipped to +- 2 * noise (via the scale parameter).
Hope this helps!


How to use FORCE INDEX or USE INDEX for query?

There are several index in my table. And i want use one of them in a specialized query.
How can i achieve that?

Did you find some way to deal with it?


try_to_load_from_cache returns cached non-existence

This PR aligns try_to_load_from_cache with its Transformers counterpart to return a special value when a file does not exist for a given repo/revision and that fact has been cached.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

(Note: related to https://github.com/huggingface/transformers/pull/18871)


500 on /v1/_ping

After upgrading our containerized private registry to 0.9 via via docker run -d --name docker-registry -v /var/lib/docker/registry:/tmp/registry -p 5000:5000 -e CORS_ORIGINS=[\'*\'] registry:0.9.0, /v1/_ping (which used to return 200/"true" on <0.9) is returning a 500.
2014-11-29 12:49:42,303 ERROR: Exception on /v1/_ping [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/app.py", line 44, in ping
    version = module.__version__
  File "/usr/lib/python2.7/dist-packages/six.py", line 116, in __getattr__
    _module = self._resolve()
  File "/usr/lib/python2.7/dist-packages/six.py", line 105, in _resolve
    return _import_module(self.mod)
  File "/usr/lib/python2.7/dist-packages/six.py", line 76, in _import_module
    __import__(name)
  File "/usr/lib/python2.7/lib-tk/tkCommonDialog.py", line 11, in <module>
    from Tkinter import *
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 42, in <module>
    raise ImportError, str(msg) + ', please install the python-tk package'
ImportError: No module named _tkinter, please install the python-tk package

After docker exec docker-registry sh -c 'apt-get update && apt-get install python-tk', I am getting another error:
9/Nov/2014:12:51:40 +0000 ERROR: Exception on /v1/_ping [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/app.py", line 44, in ping
    version = module.__version__
  File "/usr/lib/python2.7/dist-packages/six.py", line 116, in __getattr__
    _module = self._resolve()
  File "/usr/lib/python2.7/dist-packages/six.py", line 105, in _resolve
    return _import_module(self.mod)
  File "/usr/lib/python2.7/dist-packages/six.py", line 76, in _import_module
    __import__(name)
ImportError: No module named _winreg

A bit of googling seem to show that _winreg was renamed to winreg in Python 3, but:
bjaglin@caro:~$ docker exec -ti docker-registry python --version
Python 2.7.6

I reached the limits of my (inexistent) Python knowledge... Any workaround until 0.9.1 is cut out?
FWIW, our registry is behind an nginx doing HTTPS termination and basic_auth. The 500 is also thrown without proxying through nginx.

0.9.1 also not fix it.

@xiaods don't run with debug true.

@dmp42 OK

@xiaods add -e DEBUG=false or use a flavor that doesn't set debug.
Hope that helps!

@dmp42 yes, it works. but i need below function:
curl https://my.docker.registry.com/v1/_ping
true
currently it now return a {}

Yes, _ping (now) returns a json object.

@dmp42 so {} is correct? can we return some status code?

Status code is 200.

@dmp42 got it.


route one rpc to another rpc from same service

Description:
We want network devices to call rpc from one service and envoy should forward that to another rpc in same service. Message format is same.
This way we will benefit from network device sends stream different ports on Envoy but Envoy can forward it to another RPC so that the stream handling can be in separately handled.
E.g. There is grpc service StreamingUpdateService which has 3 rpcs:

PushGeneralUpdateData(stream)
PushSystemUpdateData(stream)
PushSoftwareUpdateData(stream)

Device1 calls StreamingUpdateService - PushGeneralUpdateData service on envoy port 8081 and 9081.
Listener for port 8081 should forward to StreamingUpdateService - PushSystemUpdateData
And
Listener for port 9081 should forward to StreamingUpdateService - PushSoftwareUpdateData
Not able to achieve this.
We are trying following configuration.
`admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: <IP_ADDRESS>, port_value: 9901 }
static_resources:
listeners:

name: listener_0
address:
socket_address: { address: <IP_ADDRESS>, port_value: 8081 }
filter_chains:

filters:

name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: [""]
routes:
- match: { prefix: "/StreamingUpdateService/PushGeneralUpdateData" }
route:
cluster: test_service
prefix_rewrite: "/PushSystemUpdateData"
max_grpc_timeout: 0s
cors:
allow_origin:
- ""
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
enabled: true
http_filters:

name: envoy.grpc_web
name: envoy.cors
name: envoy.router

name: listener_1
address:
socket_address: { address: <IP_ADDRESS>, port_value: 9081 }
filter_chains:

filters:

name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: [""]
routes:
- match: { prefix: "/StreamingUpdateService/PushGeneralUpdateData" }
route:
cluster: test_service
prefix_rewrite: "/PushSoftwareUpdateData"
max_grpc_timeout: 0s
cors:
allow_origin:
- ""
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
enabled: true
http_filters:

name: envoy.grpc_web
name: envoy.cors
name: envoy.router
clusters:

name: test_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
hosts: [{ socket_address: { address: <IP_ADDRESS>, port_value: 50051 }}]

`
I may be doing something wrong here.

I was able to get it working. I was using incorrect prefix rewrite options.


Helm: Improved description for tunnel, tunnelProtocol, routingMode flags

Improved description for tunnel, tunnelProtocol, routingMode flags to make it clearer for users to know the possible values.
Added deprecation note for the tunnel flag to be in line with the v1.14.0 release notes and #24561.

Please ensure your pull request adheres to the following guidelines:

[X] All commits are signed off. See the section Developer’s Certificate of Origin
[X] Provide a title or release-note blurb suitable for the release notes.

Helm: Improved description for tunnel, tunnelProtocol, routingMode flags

Internal Slack discussion: https://isovalent.slack.com/archives/C01742ANV9P/p1693834569477619

Ugh, I forgot about this, thanks @gandro! I just run this command and git amend'ed the generated file.

Multi-Pool hit   ❌ Error finalizing 'client-egress-expression-knp': timed out removing policies on Cilium agents: error dialing backend: dial tcp <IP_ADDRESS>:10250: i/o timeout which looks unrelated, but might be a new flake
conformance k8s seems to have hit https://github.com/cilium/cilium/issues/25655

Restarting both

/test

Rebased & fixed two word spellings.
@gandro, @nathanjsweet do you think we could merge this PR anytime soon? Let's please backport this to 1.14. For 1.15, tunnel should be removed anytime soon, I guess.

The main reason why this was not yet merged is that there is an outstanding review request for docs by @qmonnet - Quentin, mind taking a look? Otherwise I think we can merge without docs approval, since the changes mainly affect the Helm README.
Let's also run CI and then this should be mergable.

/test

(IPSec upgrade)[https://github.com/cilium/cilium/actions/runs/6336305356/job/17209116064] failed with known and unrelated flake https://github.com/cilium/cilium/issues/28088