Reuse conjugate gradient in newton-cg

Scipy has a conjugate gradient solver for linear equations, scipy.sparse.linalg.cg. This could be used in the optimization method "Newton-CG" of scipy.optimize.minimize which currently reimplements CG.

Related/same issue: https://github.com/scipy/scipy/issues/8792

We took a look at it.
The lines are at https://github.com/scipy/scipy/blob/main/scipy/optimize/_optimize.py#L2051 and we compared the cg implementation with the implementation of scipy.sparse.linalg.cg at  https://github.com/scipy/scipy/blob/main/scipy/sparse/linalg/_isolve/iterative.py#L308 ,
but i dont think we can just easily replace it by scipy.sparse.linalg.cg, since there are some optimizations for the matrix-vector product, with the hessian evaluations.

My background is more numerical linear algebra, but my understanding is that although the CG iterations in Newton-CG are similar to those in the CG implementation, Newton-CG needs to handle negative curvature differently (e.g., the presence of negative eigenvalues). With standard CG, the algorithm will fail if the matrix is not symmetric positive definite. However, with Newton-CG even if the Hessian is indefinite (i.e., has negative curvature), we can still extract a useful descent direction from the current iterate of CG; note the break when negative curvature is detected here.
With a bit of work, we could combine the CG component of Newton-CG with the pure CG implementation, but we would need add logic to return the partial progress in the case of negative curvature. My instinct is that combining ~5 lines of the two CG implementations doesn't merit the additional complexity, but that is up to the maintainers to decide.
Looking over the code though, we may want to reconsider some of the termination conditions. For example, the default allows 20*n iterations of CG, where n is the dimension of the Hessian---this may lead to too much work finding a descent direction when taking an approximate step would be preferable. @ccmagruder Do you have any insights?

Good arguments against.


[Q] Compatibility with Qt commercial qtcreator

I'm trying to test if I can make app with qtcreator and install via appman to the pelux.
I installed qtcreator(commercial trial) which provides 'automotive suites examples'.
First of all, I configured the pelux toolchain kit (for rpi-auto) as described in pelux doc
Then I selected a example in the qtcreator, which named 'hello-world'.
The sample just  has only qml files without native C/C++ file.
Since the sample consists of qml files only, it doesn't need link step. But generated Makefile by qmake has the following line.
$(TARGET):  $(OBJECTS)
    $(LINK) $(LFLAGS) -o $(TARGET) $(OBJECTS) $(OBJCOMP) $(LIBS)

However if I did with pre-existing automotive-suite toolchain included in the qt installer (for intel NUC), the generated Makefile was slight different.
hello-world:  $(OBJECTS)

Eventually, it fails on 'link' step when I try with pelux toolchain. But with 'intel NUC' automotive-toolchain, there is no issue.
Do you know why different Makefiles are created based on the same qmake project file?

Thanks @Gagi2k .
I have some questions for the qt designer.
By default, in order to view the qml file in the designer view, does the qml plugins which in the qml file need to be loadable on the PC?
If the qml plugin is cross-compiled only, it seems not be shown on the designer view.
And some qml plugins which are only loadable in the certain environment like wayland.(i.g. ApplicationManagerWindow).  These plugin types seem not be shown on the designer view regardless it is compiled for x86 or target.
If I misunderstood, please let me know.

Sorry i don't know, i never used the QML form editor

@sparkleholic No, you are fully correct. The "Designer" in Qt Creator was initially not designed for non-desktop environments which could use custom Window types. What surprises more is that the new Qt Design Studio does not consider this either whereas it was made in awareness of embedded systems which might not be based on the standard Window type. I've created a bug report to clarify this: https://bugreports.qt.io/browse/QDS-511


rename flow_go commands to something more intuitiveIssue Title

The 2 variations of the FLOW_GO command that start and end a flow script are confusing, and more complicated then they need to be. Need to implement improved names, and update the help command and the creating-fows documentation

how about replacing FLOWGO, 0, 0   with NEW_FLOW  , shortform NF, with the syntax being just the command for now
replace FLOW_GO,1,msecPerFrame, toe-action   with DO_FLOW , shortform DF.
Syntax: DF, toe-action, msecPerFrame
changing the order lets you leave out the msecPerFRame,  to specify a toe-action and the default msecPerFRame value.

Also need to modify all existing scripts to use the new commands and syntax


kube_elasticsearch: Deploy Cerebro

Fixes: #126
See: https://github.com/scality/metal-k8s/issues/126

Not ready for merge, I'm working on docs.
Indeed, I want to do a PR against Charts to be able to add annotations, and
specify a config file or hosts list etc. The current chart is a bit lacking.
On Fri, 6 Jul 2018, 13:33 Zempashi<EMAIL_ADDRESS>wrote:

@Zempashi approved this pull request.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/scality/metal-k8s/pull/127#pullrequestreview-134980618,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGQzeRpL5HPKSFWcYvBAJhDxW3ZZuzkks5uD0sEgaJpZM4VFNKO
.

--
https://www.scality.com/download-idc-marketscape/?utm_source=ScalityEmail&utm_medium=Email&utm_campaign=2018Predictions

Docs done, ready for review. Adding service annotations will take upstream changes, will work on those but not strictly required before merge.

Follow-up: https://github.com/kubernetes/charts/pull/6595


AJV validator and custom or user-friendly error message

I have the following schema and json to validate using ajv. I am developing a REST API that takes a JSON and gets validated against the schema and it returns the error (400- with the ajv error) or (200 - when successfully validated)
//Assuming ajv, ajv-errors npm are installed properly
const schema = {
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": [ "countries" ],
  "definitions": {
    "europeDef": {
      "type": "object",
      "required": ["type"],
      "properties": { "type": {"const": "europe"} }
    },
    "errorMessage":"Country type should be europe only"
    "asiaDef": {
      "type": "object",
      "required": ["type"],
      "properties": { "type": {"const": "asia"} }
    },
    "errorMessage":"Country type should be asia only"
  },
  "properties": {
    "countries": {
      "type": "array",
      "items": {
        "oneOf":[
          { "$ref": "#/definitions/europeDef" },
          { "$ref": "#/definitions/asiaDef"}
        ]
      }
    }
  }
}

const data = {
  "countries":[
    {"type": "asia1"},
    {"type": "europe"}
  ]
}

const isValid = ajv.validate(schema, data); //schema, data
if(! isValid){
  console.log(ajv.errors);
}

and the error is:
[
  {
    keyword: 'errorMessage',
    dataPath: '/countries/1',
    schemaPath: '#/definitions/europeDef/errorMessage',
    params: { errors: [Array] },
    message: 'Country type should be europe only'
  },
  {
    keyword: 'errorMessage',
    dataPath: '/countries/1',
    schemaPath: '#/definitions/asiaDef/errorMessage',
    params: { errors: [Array] },
    message: 'Country type should be asia only'
  },
  {
    keyword: 'oneOf',
    dataPath: '/countries/1',
    schemaPath: '#/properties/countries/items/oneOf',
    params: { passingSchemas: null },
    message: 'should match exactly one schema in oneOf'
  }
]

I know why the error is appearing (reason: as I have used 'asia1' and it is not conforming the schema standard). Here, the problem is with "asia", but the ajv is giving the error mesage for 'europe' too (despite it is correct)
My question is, as I have derived this schema so I can pretty much understand the error. But for a third person it would definitely take some time to figure it out (and it may take more time, if the schema/errors are more complex).
If I returned that whole error message as a response as it is, it will be more complex error message to understand and to present to the enduser.
So, Is there any way by which I can provide more meaningful & user friendly/specific error message to understand using this ajv-errors npm? I tried but not able to make it possible using this npm.
ex: Invalid asia1 found for country type found.
P.S: Originally asked at SoF: Link and slightly updated for this issue in this git repo.

You can use if/then/else or custom select (from ajv-keywords) to avid validating branches that are known to be invalid because of the type tag. There is an ticket in Ajv to change oneOf validation to use discriminator keyword to achieve the same behaviour, but it is not implemented yet.


Syntax for 64-bit keywords

To specify a 64-bit keyword, use a capital X in your number, e.g.
-matchanykw 0Xffffffff1234abcd
You must do this for manifested ETW providers.
If you use a lower case 'x', e.g. 0x12345678, it will parse it as a 32-bit number.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 82f883e7-ff30-ab48-56d5-a9678445f7d4
Version Independent ID: 0852c3d6-ff1d-1650-c43e-d3ea145cad0c
Content: Tracelog Command Syntax - Windows drivers
Content Source: windows-driver-docs-pr/devtest/tracelog-command-syntax.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @EliotSeattle
Microsoft Alias: eliotgra

Oops, that's incorrect. Ignore this.


firebase_ml_vision supports FirebaseVisionImage.fromBytes() without FirebaseVisionImageMetadata

Use case
Given an Uint8List photo, we have to store to a local file in order to convert FirebaseVisionImage
There is a FirebaseVisionImage.fromBytes() function, but it perhaps is for CameraImage only, with the normal photo such retrieving from the Internet, we have to store to a file and then use FirebaseVisionImage.fromFile() that wastes an additional step and space.
Proposal
Update this function
FirebaseVisionImage.fromBytes(
    Uint8List bytes
  )

With FirebaseVisionImageMetadata is optional

hI @thanhbinh84
From what I can see, the issue is related to a FlutterFire plugin rather than to Flutter itself. Please open the issue in the dedicated repository.
Closing, as this isn't an issue with Flutter itself.
If you disagree, please write in the comments,
providing your flutter doctor -v, your flutter run -v,  your pubspec.yaml
a minimal reproducible code sample that does not use 3rd party plugins,
and I will reopen it.
Thank you


How to compile?

As i was compiling this code, i got a bunch of errors. I Have no experience with rust so please help me out :)

Hello @d4r8n, thanks for opening up this issue.
Instructions for compilation
1 - Install VSBuildTools.
2 - Install Rustup.
3 - Ensure Rustup is configured to use build tools, by running the following in powershell/cmd:
rustup default stable-x86_64-pc-windows-msvc

4 - Navigate to the root directory of the project, where Cargo.toml is, and you can run the following in powershell/cmd:
cargo build --release # Build the binary with optimization, saved to /target/release/[program].exe
cargo build # Build the binary without optimization, saved to /target/debug/[program].exe

cargo run --release # Run the binary with optimization
cargo run # Run the binary without optimization

If it still doesn't work
Send any error messages, or logs, here and I will attempt to diagnose the issue.


Feature request: Append content to auth page

There is a slot to replace the whole auth page

Authentication

Replaces the authentication section

However if I want to make use of the default auth token input, I'd love to just be able to add content below and or above the default page

Could be interesting. What are you thinking about for content there. If there is something useful there we could just directly add it into the library. What sort of content would you like on this tab?

Thanks for the speedy reply!
I'd generally like to describe the behavior's of the token that is used for authentication, and probably also link to / explain how to generate the token.
Out of the box it looks a little bare, and as the menu item is called "authentication" I'd expect this is where people will end up clicking to try to learn how to authenticate

How complex is your authentication? For instance at Authress, we replaced the Authentication with this: https://authress.io/app/#/api?route=auth
Then we used the events to automatically inject in the necessary token.
You can also dynamically set the token using the authentication library api So you can make this whole part look exactly like you want and still have the hooks to the functionality.
I don't know what your authentication flow looks like or which one you are using, but if it is really just the API key one, then using the function setAuthenticationConfiguration() with a custom UI component might be the best for your use case.
If that doesn't work then we can try to figure out if there is a slot that makes sense to inject here.
As an aside, I do want to caution that pushing the API keys into the UI is a bit unsafe. And depending on how you are generating the API keys they could be even more potential issues with that. I don't know if you are interesting in some relevant docs about that, but security of apis is my specialty. FWIW, Bearer JWTs is going to likely be a much safer option for your users/service clients.

https://authress.io/app/#/api?route=auth looks like exactly what I want to aim for.
I guess I was just trying to avoid having to reimplement the input box and having it tie into the rendered sample requests and of course the actual requests getting fired off.
So I'll go ahead and give setAuthenticationConfiguration a go with a custom page.
Any docs you think may be of interest to me I'd love to give a once over :)
We will be using bearer JWTs, and if any auto generated tokens are used in the UI I expect I'll end up creating them with a rather short TTL lifespan

FWIW, if you inject the token using either function--i.e. the curl request OR the setAuthenticationConfiguration--then the full curl will contain the relevant information. it is dynamically updated.
For instance, this is literally our requestInterceptor
requestInterceptor(event) {
      if (this.apiToken) {
        event.detail.request.options.headers.append('Authorization', this.apiToken);
      }

      if (event.detail.request.explorerLocation === 'get-/' || event.detail.request.explorerLocation === 'post-/api/authentication/oauth/tokens' && event.detail.request.url.match('authress.io')) {
        this.showOauthPopupDetails = true;
        return;
      }
    },

with (this is vue):
<openapi-explorer table collapse v-if="openapiSpecificationUrl" :explorer-location="explorerLocation"
:spec-url="openapiSpecificationUrl" @request="requestInterceptor" @response="responseInterceptor" />

Same goes for the setAuthenticationConfiguration, there's an example in the docs:
https://github.com/Rhosys/openapi-explorer/blob/5595a4617019e27259f7005b290e3ebd37138619/docs/documentation.md?plain=1#L44
and a usage in the mocks: https://github.com/Rhosys/openapi-explorer/blob/5595a4617019e27259f7005b290e3ebd37138619/mocks/index.html#L53
If none of those help, just let us know.

We will be using bearer JWTs, and if any auto generated tokens are used in the UI I expect I'll end up creating them with a rather short TTL lifespan

:+1:, if you run into any problems there, feel free to let us know, since, um, we have a product specifically for that :wink:.
With that, I'll close this ticket for now.


Convert NXP Kinetis Pinmux to get info from devicetree (for device objects, not pins)

This is based on how the rv32m1 pinmux driver & board code works.


Tag v2 Release

Is v2 pre release? The docs website leads me to believe that v2 has been released. However in order to install v2 I must composer require mateusjunges/laravel-trackable-jobs:dev-master
Thanks in advance.

I'm also waiting for a v2 release 🙏

hey yall 👋 I did not had the time to finish the what's needed to release a v2 yet. Been working on other projects for a while. I'll see if I can make it happen within a month.

Alright, I worked on it this weekend and decided on a few things:

Jobs will be tracked even if they implement the ShouldBeUnique interface and are not queued by laravel. They will remain in the database with the status of created.
Dispatch without tracking feature will be dropped as this is really hard to maintain, and neither me or the guy who requested this feature are using it atm. If anyone has a good solution on how to solve it, feel free to open a PR.
V2 will use a base class instead of the Trackable trait. You'll have to extend your job from the TracableJob class in order to add tracking behavior

let me know what yall think. I'll probably release v2 this week.

Beta 1 released. Appreciate any feedback
https://github.com/mateusjunges/trackable-jobs-for-laravel/releases/tag/v2.0.0-beta.1

ok folks, v2 just dropped. Closing this


[FEATURE] minio and s3, as a file search alternative

Describe the feature you'd like
Something more robust would be of great value to the system, both for saving and retrieving documents.
An alternative to this can be minio (an open source based on amazon's s3) and s3 itself

welcome any contribution to more doc loaders like S3 - https://js.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_loaders/s3 !


Export Schnorr signature API

Tested against BIP340 reference test vectors and, through https://github.com/ACINQ/bitcoin-kmp/pull/40, against "key path spending" BIP341 test vectors.


Update Post Scriptum

Game Title
Post Scriptum
Status of the Anti-Cheat
🚫 Denied
Tinkering Steps

Proof of Mention
see screenshots

Also, it's worth noting that you CAN play Post Scriptum online without EAC, but the server has to disable EAC. The game works perfectly on Linux, other than the anti-cheat. I have personally verified this by running my own server with anti-cheat disabled, and joining it along with some friends, and randoms from the server browser.


log_backup: fix panic when encountered error during resuming

What is changed and how it works?

Issue Number: Close #17020

What's Changed:
Spawn thread from the thread pool directly. (Instead of the thread local runtime handle.)

Related changes

[ ] PR to update pingcap/docs/pingcap/docs-cn:
[ ] Need to cherry-pick to the release branch

Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Release note

Fixed a bug that might cause TiKV panic when resuming a PiTR task and the connection to PD isn't stable enought.

/merge


Data loss while writing avro file to s3 compatible storage

Hi,
I am converting a csv file into avro and writing to s3 compliant storage.I see that schema file(.avsc) is written properly. However, there is data loss while writing to .avro file.
Below is snippet of my code
## Code
import smart_open
from boto.compat import urlsplit, six
import boto
import boto.s3.connection

import avro.schema
from avro.datafile import  DataFileWriter
from avro.io import  DatumWriter

import pandas as pn
import os,sys

FilePath = 's3a://mybucket/vinuthnav/csv/file1.csv' #path on s3

splitInputDir = urlsplit(FilePath, allow_fragments=False)

inConn = boto.connect_s3(
	aws_access_key_id = access_key_id,
	aws_secret_access_key = secret_access_key,
	port=int(port),
	host = hostname,
	is_secure=False,
	calling_format = boto.s3.connection.OrdinaryCallingFormat(),
	)
#get bucket
inbucket = inConn.get_bucket(splitInputDir.netloc)
#read in the csv file
kr = inbucket.get_key(splitInputDir.path)
with smart_open.smart_open(kr, 'r') as fin:
	xa = pn.read_csv(fin, header=1, error_bad_lines = False).fillna('NA')

rowCount, columnCount = xa.shape #check if data frame is empty, if it is, don't write outp
if rowCount == 0:
	##do nothing
	print '>> [NOTE] empty file'

else:
	#generate avro schema and data

	dataFile = os.path.join(os.path.basename(FileName), os.path.splitext(FileName)[0]+".avro")
	schemaFile = os.path.join(os.path.basename(FileName), os.path.splitext(FileName)[0]+".avsc")

	kwd = inbucket.get_key(urlsplit(dataFile, allow_fragments=False).path, validate=False)
	schema = gen_schema(xa.columns)

	with smart_open.smart_open(kwd, 'wb') as foutd:

		dictRes = xa.to_dict(orient='records')
		writer = DataFileWriter(foutd, DatumWriter(), schema)
		for ll, row in enumerate(dictRes):
			writer.append(row)

Thank you for providing example source. Could you please describe what the problem is in detail? An example of the data loss, as well as how you detected it, would be very helpful.

I converted the avro file to Json and read the number of lines. I expect the number of lines in Json to match the number of lines in csv and it didn't match.

Have you confirmed that this problem still persists if you use local storage instead of smart_open?

Yes, this problem doesn't exist if I use local storage . However file on local storage was not written through smart_open

Yes file is being read correctly. I counted number of lines in the dataframe i loaded . i also counted the datumWriter count. It matches but after writing to avro on s3 and reading it i see there is data loss. This confirms that problem is with writing binary data to s3

What version of smart_open are you using? Have you observed a similar problem with older versions?

smart_open-1.3.5 with anaconda version of python 2.7

I haven't tried with other versions

OK, thank you for providing detailed information about this bug. We will investigate.

thanks

Hi, do you have any update?

Sorry, no, I have not looked into this yet. Is this urgent for you?

Yes, this is a blocker for us . We are also searching some alternative to smart open. Haven't found any yet. At least a work around asap would be great

I think the workaround is to write the avro file to local storage first, and upload to S3 when it is complete.
I've started investigating the issue, but cannot reproduce the problem because your code sample is incomplete: the gen_schema function is missing. Could you please look at this file:
https://github.com/mpenkov/smart_open/blob/209/integration-tests/test_209.py
and update it so that it reproduces your problem?

Where is the updated file?

I have updated the code in test_209.py you provided. It showed pushed

I think your push failed. Here’s the file in your repo: https://github.com/vinuthna91/smart_open/blob/209/integration-tests/test_209.py
It’s unchanged from my version.
If you cant work it out, please paste the code here as a comment.

I changed it, I am sorry for delayed response. I was out of station.
https://github.com/vinuthna91/smart_open/blob/209/integration-tests/test_209.py
def gen_schema(paramNames):
paramNamesLen = len(paramNames)
dataName = 'schema'
avroSchemaOut = "{\n\t\"type\": 	\"record\", \"name\": \"%s\", \"namespace\": \"com.sandisk.bigdata\", \n \t\"fields\": [" %(dataName)

if paramNamesLen==0:
	#no parameters, no schema file generation
	avroSchemaOut = ''

else:
	#generate file
	for ii in range(paramNamesLen):
		typeString = "[\"%s\", \"null\"]" %('String')
		schemaString = "{ \"name\":\"%s\", \"type\":%s, \"default\":null}" % (paramNames[ii], typeString)
		if ii == 0:
			avroSchemaOut += schemaString + ',\n'
		elif ii <len(paramNames)-1:
			avroSchemaOut += "\t\t\t" + schemaString + ',\n'
		else:
			avroSchemaOut += "\t\t\t" + schemaString + '\n'
	avroSchemaOut += "\n \t\t\t]\n}"

return avroSchemaOut

I pulled your changes and made some updates. Unfortunately, the code crashes with an error unrelated to smart_open:
  File "integration-tests/test_209.py", line 61, in <module>
    writer.append(row)
  File "/Users/misha/envs/smartopen27/lib/python2.7/site-packages/avro/datafile.py", line 196, in append
    self.datum_writer.write(datum, self.buffer_encoder)
  File "/Users/misha/envs/smartopen27/lib/python2.7/site-packages/avro/io.py", line 768, in write
    if not validate(self.writers_schema, datum):
  File "/Users/misha/envs/smartopen27/lib/python2.7/site-packages/avro/io.py", line 103, in validate
    schema_type = expected_schema.type
AttributeError: 'str' object has no attribute 'type'

Please have a look and update the code. The updated code is here: https://github.com/mpenkov/smart_open/blob/209/integration-tests/test_209.py

Updated code is here:
https://github.com/mpenkov/smart_open/pull/1/commits/3eaf46f0ca48d23b50f15921bdb16fe812ed17e2

Thank you for the update. Unfortunately, I still cannot reproduce your problem because of errors in avro.
  "namespace": "com.sandisk.bigdata",
  "type": "record",
  "name": "schema",
  "fields": [
    {
      "default": null,
      "type": [
        "null",
        "string"
      ],
      "name": "15109264"
    },
    {
      "default": null,
      "type": [
        "null",
        "string"
      ],
      "name": "EFILE"
    },

Can you please make sure the test runs and reproduces your actual problem?

Can you give the csv you used?

The CSV gets downloaded as part of the actual test. Please see the source
file.
On Thu, Jul 26, 2018 at 20:46 vinuthna91<EMAIL_ADDRESS>wrote:

Can you give the csv you used?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/RaRe-Technologies/smart_open/issues/209#issuecomment-408070182,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABDOVNa4S6NuvZly7GhcHDkbWl-zmRhYks5uKawQgaJpZM4VPAWw
.

Could you please check now. I updated the code.
smart_open/integration-tests/test_209.py

Were you able to try it? Please confirm that the code is working now

Yes, the code is working now. I was able to use it to reproduce the problem. I also simplified it a bit to rule out errors in the actual test: https://github.com/mpenkov/smart_open/blob/209/integration-tests/test_209.py
I'll take it from here. Thanks for your help in reproducing the problem.

$ SO_NUMROWS=10000 SO_S3_URL=s3://dbi2-ue2/test_smart_open python integration-tests/test_209.py
download: s3://dbi2-ue2/test_smart_open/issue_209/out.avro to ./remote.avro
Binary files local.avro and remote.avro differ

The two avro files are:

local.avro: written straight to local storage
remote.avro: written to S3 storage via smart_open, and then retrieved via the CLI
They should be the same, but they are different for some reason, suggesting there is a problem with smart_open.

Some more details:
(smart_open2.7) misha@cabron:~/git/smart_open$ SO_NUMROWS=477 SO_S3_URL=s3://dbi2-ue2/smart_open_test/ python integration-tests/test_209.py
CRITICAL:smart_open.s3:wrote 8192 bytes (8192 total)
CRITICAL:smart_open.s3:wrote 8192 bytes (16384 total)
CRITICAL:smart_open.s3:wrote 8192 bytes (24576 total)
CRITICAL:smart_open.s3:wrote 8192 bytes (32768 total)
CRITICAL:smart_open.s3:wrote 8192 bytes (40960 total)
CRITICAL:smart_open.s3:wrote 8192 bytes (49152 total)
CRITICAL:smart_open.s3:wrote 8192 bytes (57344 total)
CRITICAL:smart_open.s3:wrote 7473 bytes (64817 total)
download: s3://dbi2-ue2/smart_open_test//issue_209/out.avro to ./remote.avro
sanity check OK
CRITICAL:smart_open.s3:wrote 4 bytes (4 total)
CRITICAL:smart_open.s3:wrote 1 bytes (5 total)
CRITICAL:smart_open.s3:wrote 1 bytes (6 total)
CRITICAL:smart_open.s3:wrote 11 bytes (17 total)
CRITICAL:smart_open.s3:wrote 1 bytes (18 total)
CRITICAL:smart_open.s3:wrote 1 bytes (19 total)
CRITICAL:smart_open.s3:wrote 680 bytes (699 total)
CRITICAL:smart_open.s3:wrote 1 bytes (700 total)
CRITICAL:smart_open.s3:wrote 10 bytes (710 total)
CRITICAL:smart_open.s3:wrote 1 bytes (711 total)
CRITICAL:smart_open.s3:wrote 4 bytes (715 total)
CRITICAL:smart_open.s3:wrote 1 bytes (716 total)
CRITICAL:smart_open.s3:wrote 16 bytes (732 total)
CRITICAL:smart_open.s3:wrote 1 bytes (733 total)
CRITICAL:smart_open.s3:wrote 1 bytes (734 total)
CRITICAL:smart_open.s3:wrote 1 bytes (735 total)
CRITICAL:smart_open.s3:wrote 1 bytes (736 total)
CRITICAL:smart_open.s3:wrote 1 bytes (737 total)
CRITICAL:smart_open.s3:wrote 64064 bytes (64801 total)
CRITICAL:smart_open.s3:wrote 16 bytes (64817 total)
download: s3://dbi2-ue2/smart_open_test//issue_209/out.avro to ./remote.avro
Binary files local.avro and remote.avro differ
test NG
(smart_open2.7) misha@cabron:~/git/smart_open$ hexdump local.avro > local.hex
(smart_open2.7) misha@cabron:~/git/smart_open$ hexdump remote.avro > remote.hex
(smart_open2.7) misha@cabron:~/git/smart_open$ diff --suppress-common-lines --side-by-side remote.hex local.hex
00002c0 632e 646f 6365 6e08 6c75 006c c515 49a0               | 00002c0 632e 646f 6365 6e08 6c75 006c db3c 70da
00002d0 35ce aed7 8e65 0591 4a65 5a8e 07ba e980               | 00002d0 4162 c706 555f 8eb3 987a 395b 07ba e980
000fd20 1535 a0c5 ce49 d735 65ae 918e 6505 8e4a               | 000fd20 3c35 dadb 6270 0641 5fc7 b355 7a8e 5b98
000fd30 005a                                                  | 000fd30 0039

The same number of bytes get written in both cases, but there seems to be some corruption happening from bytes 02c0 to 02e0 (704 to 736 in decimal); and fd20 to fd40 (64800 to 64832 in decimal).
Interesting points:

The first badness includes several write boundaries.
The second badness is in the last 32 bytes on the file. It includes a boundary between writes.

@vinuthna91 I've had a look at the issue. It appears avro is sending slightly different data when writing locally compared to when writing to S3 via smart_open. Have a look at this branch.
I've also added logging to the library in the critical places. This logging is important for demonstrating the problem.
Now, when we run the test, we get this:
(smart_open2.7) misha@cabron:~/git/smart_open$ SO_NUMROWS=477 SO_S3_URL=s3://dbi2-ue2/smart_open_test/ python integration-tests/test_209.py
CRITICAL:root:writing to <_io.BufferedWriter name='local.avro'>
CRITICAL:smart_open.smart_open_lib:wrote 10 bytes (710 total)
CRITICAL:smart_open.smart_open_lib:6176 726f 2e63 6f64 6563
CRITICAL:smart_open.smart_open_lib:wrote 1 bytes (711 total)
CRITICAL:smart_open.smart_open_lib:08
CRITICAL:smart_open.smart_open_lib:wrote 4 bytes (715 total)
CRITICAL:smart_open.smart_open_lib:6e75 6c6c
CRITICAL:smart_open.smart_open_lib:wrote 1 bytes (716 total)
CRITICAL:smart_open.smart_open_lib:00
CRITICAL:smart_open.smart_open_lib:wrote 16 bytes (732 total)
CRITICAL:smart_open.smart_open_lib:83a8 3a59 9a9c 0ba4 b495 9802 bd78 ef59
CRITICAL:smart_open.smart_open_lib:wrote 1 bytes (733 total)
CRITICAL:smart_open.smart_open_lib:ba
CRITICAL:smart_open.smart_open_lib:wrote 1 bytes (734 total)
CRITICAL:smart_open.smart_open_lib:07
CRITICAL:smart_open.smart_open_lib:wrote 1 bytes (735 total)
CRITICAL:smart_open.smart_open_lib:80
CRITICAL:root:writing to <smart_open.s3.BufferedOutputBase object at 0x7fdc65025310>
CRITICAL:smart_open.s3:wrote 10 bytes (710 total)
CRITICAL:smart_open.s3:6176 726f 2e63 6f64 6563
CRITICAL:smart_open.s3:wrote 1 bytes (711 total)
CRITICAL:smart_open.s3:08
CRITICAL:smart_open.s3:wrote 4 bytes (715 total)
CRITICAL:smart_open.s3:6e75 6c6c
CRITICAL:smart_open.s3:wrote 1 bytes (716 total)
CRITICAL:smart_open.s3:00
CRITICAL:smart_open.s3:wrote 16 bytes (732 total)
CRITICAL:smart_open.s3:557a 9c7f 1ee2 88eb 41bf 0dfb 880d b30e
CRITICAL:smart_open.s3:wrote 1 bytes (733 total)
CRITICAL:smart_open.s3:ba
CRITICAL:smart_open.s3:wrote 1 bytes (734 total)
CRITICAL:smart_open.s3:07
CRITICAL:smart_open.s3:wrote 1 bytes (735 total)
CRITICAL:smart_open.s3:80
download: s3://dbi2-ue2/smart_open_test//issue_209/out.avro to ./remote.avro
Binary files local.avro and remote.avro differ
test NG
(smart_open2.7) misha@cabron:~/git/smart_open$

There are two separate writes happening here:

avro writes to local storage via smart_open
avro writes to S3 via smart_open

Have a look at the bytes that avro is passing to smart_open. In the first case:
CRITICAL:smart_open.smart_open_lib:wrote 16 bytes (732 total)
CRITICAL:smart_open.smart_open_lib:83a8 3a59 9a9c 0ba4 b495 9802 bd78 ef59

and in the second case:
CRITICAL:smart_open.s3:wrote 16 bytes (732 total)
CRITICAL:smart_open.s3:557a 9c7f 1ee2 88eb 41bf 0dfb 880d b30e

The bytes are different.
Therefore, I think this may be a problem in avro, not smart_open. @menshikh-iv Please have a look and double-check my reasoning. It's also possible I messed something up in the code, so please have a look at that, too.

Avro takes foutd as argument which is file writer. For local storage it is just fopen argument, while in s3 case, it is generated from smart open isn't it? Do you think avro cannot generate correct data for smart open key?

Could you suggest any other way of writing avro files to s3 using python instead of using avro datum writer?

Do you think avro cannot generate correct data for smart open key?

I'm not 100% sure, but that's the only plausible explanation I can offer, given the test results above. To be 100% sure of what's going on, you'd have to debug avro, confirm that it's sending different data, and look for a reason why.

Could you suggest any other way of writing avro files to s3 using python instead of using avro datum writer?

Sorry, I'm entirely unfamiliar with avro.
I have a follow-up question for you. How do you convert the avro files to JSON, so that you can see the effect of data corruption?

I have a follow-up question for you. How do you convert the avro files to JSON, so that you can see the effect of data corruption?
I use a Java jar to do this. Command below-
java -jar avro-tools-1.8.2.jar tojson <avrofilename.avro> > jsonfilename.json

@scottbelden Thank you for looking into this.
S3 is a cloud-based storage solution from Amazon Web Services. S3 stores data in collections called buckets. In your case, you're trying to write to a bucket that you don't have write access to, as the error at the end of the stack trace is trying to tell you.
To help you solve the problem, I've added the option to mock out S3. This way, you can reproduce the original problem without having write access to an S3 bucket. Please do a pull from my 209 branch, and then run:
$ SO_ENABLE_MOCKS=1 SO_NUMROWS=477 SO_S3_URL=s3://dummy python integration-tests/test_209.py

OK, then we need to make the tests more robust. Can fastavro convert avro files to JSON? I'd rather not have to include java in the test to do the same thing.

I have a follow-up question for you. How do you convert the avro files to JSON, so that you can see the effect of data corruption?
I use a Java jar to do this. Command below-
java -jar avro-tools-1.8.2.jar tojson <avrofilename.avro> > jsonfilename.json

Converting to JSON is something I'm actually working on in my free time at the moment, but it doesn't quite work yet. However, you can use the regular avro library to read the file and spit out the list of records and compare those.
To read the records you can do this:
def read_avro(foutd):
    reader = avro.datafile.DataFileReader(foutd, avro.io.DatumReader())
    return list(reader)

And then you could add some check like you have with the binary diff:
with open('local.avro', 'rb') as fout:
    local_records = read_avro(fout)

with open('remote.avro', 'rb') as fout:
    remote_records = read_avro(fout)

if local_records == remote_records:
    print('Same')
else:
    print('Different')

When I do that I get:
Same
Binary files local.avro and remote.avro differ
test NG

@scottbelden Thank you for providing insight about avro. I've updated the test.
@vinuthna91 Please see the updated test. Everything passes, so we cannot reproduce your error anymore. We get the same result with and without mocking.
Please try to reproduce the error using the updated test code.
CRITICAL:root:writing to <open file 'local.avro', mode 'wb' at 0x7f87d5c60150>
CRITICAL:root:writing to <_io.BufferedWriter name='local-so.avro'>
sanity check OK
CRITICAL:root:output_url: 's3://dummy/issue_209/out.avro' bucket_name: 'dummy' key_name: 'issue_209/out.avro'
CRITICAL:root:writing to <smart_open.s3.BufferedOutputBase object at 0x7f87d21bfbd0>
test OK
binary check OK
(smart_open2.7) misha@cabron:~/git/smart_open$ SO_S3_URL=s3://dbi2-ue2/smart_open_test python integration-tests/test_209.py
CRITICAL:root:writing to <open file 'local.avro', mode 'wb' at 0x7fa6594cc150>
CRITICAL:root:writing to <_io.BufferedWriter name='local-so.avro'>
sanity check OK
CRITICAL:root:output_url: 's3://redacted/smart_open_test/issue_209/out.avro' bucket_name: 'dbi2-ue2' key_name: 'smart_open_test/issue_209/out.avro'
CRITICAL:root:writing to <smart_open.s3.BufferedOutputBase object at 0x7fa65596d310>
test OK
binary check OK
(smart_open2.7) misha@cabron:~/git/smart_open$

so, fix is to set the avro sync marker to same value to avoid data loss?
@mock.patch('avro.datafile.DataFileWriter.generate_sync_marker', mock.Mock(return_value=b'0123456789abcdef'))

No, that line is there so that the sync marker does not interfere with the binary check. You may remove it during your tests, but if you do so, be sure to disable the binary check at the end as well.
The current test results show that there is no data loss. Please confirm using your own data and let me know if you get a different result.

Ok, I will check and let you know

I checked on my dataset. local.avro and remote.avro matched but total number of records in avro file and csv file did not match even by reading with read_avro function in your code.

number of records read by avro should be same as number of data lines

If local.avro and remote.avro matched, the problem isn't in smart_open.

@mpenkov  thanks for the help. I will debug more to see what is the issue in the script.

@mpenkov I have an observation:
while i read csv from local storage, convert it to avro
local.avro and remote.avro do not differ.
However, i read csv from s3, convert it to avro
local.avro and remote.avro differ.

Without seeing your actual test case (source code or at least shell commands), it's hard for me to comment.
I ran an additional test to ensure that there is no data loss when reading files from S3:
(smart_open2.7) misha@cabron:~/git/smart_open$ python integration-tests/s3_cp.py s3://irs-form-990/index_2018.csv index_2018_smartopen.csv
(smart_open2.7) misha@cabron:~/git/smart_open$ aws s3 cp s3://irs-form-990/index_2018.csv index_2018.csv
download: s3://irs-form-990/index_2018.csv to ./index_2018.csv
(smart_open2.7) misha@cabron:~/git/smart_open$ diff index_2018_smartopen.csv index_2018.csv
(smart_open2.7) misha@cabron:~/git/smart_open$

I downloaded the CSV file using 1) smart_open 2) AWS CLI and compared the two files. They were identical, so there is no data loss in the reading case, either.

I have used same code provided by you except that i used my local data set. My csv file size is only 90KB
On line number 72, I have added #writer.close(). When writer.close() is uncommented, writing local.avro to local storage is complete and there is no data loss. However, writing to s3 storage throws an error AttributeError: 'S3OpenWrite' object has no attribute 'flush'.
Here local.avro and remote.avro differ
When you comment this( writer.close() is not used),  local.avro and remote.avro both are same but, there is data loss in both the files when compared to csv data.
That is required to write avro completely.

I don't know how pandas works, but if you look at the data variable which is the result of running the csv file through pandas you'll notice that it only has [477 rows x 9 columns]. The csv file has over 172,000 rows, so it seems like some data is getting truncated there.

i don't see truncation in number of rows in pandas data frame. i see it only in the avro file. I am counting number of records written to local avro file when we use writer.close() and when we do not use.

When I read in all the data using SO_NUMROWS=477 and then print the resulting number or records I see that local.avro has 172396 while remote.avro has 172389

when writer.close() is called it internally calls a flush() method that Flushes the current state of the file, including metadata. The error i am getting is 'S3OpenWrite' object has no attribute 'flush'. because of which metadata is being lost.

I don't see that exception from the test script, but yeah, that would make sense.

@scottbelden  that is exactly the data loss i am talking about.
If you uncomment writer.close() in line#72, you should see the exception while writing to avro file on s3

I'm not sure which script has writer.close() on line 72...

I made the change to test script. If you do not see the change, you need to add writer.close() to the write_avro function definition:
def write_avro(foutd):
schema = avro.schema.parse(avroSchemaOut)
dictRes = data.to_dict(orient='records')
writer = avro.datafile.DataFileWriter(foutd, avro.io.DatumWriter(), schema)
for ll, row in enumerate(dictRes):
writer.append(row)
writer.close()

@vinuthna91 Why are you closing the writer manually? The writer is a context manager. It will take care of closing and cleaning up when the context exits (at the end of the "with" block).

Thanks!

could you find anything?

OK, I think see what's going on here. I'll look at this again on the weekend.
It's still Thursday. Weekend starts in two days.

Okay.

@mpenkov Thanks for looking into it.
unfortunately, i used the same code and it still throws me error about absence of flush method.
Error below-
CRITICAL:root:writing to <open file 'local.avro', mode 'wb' at 0x7fdb98674ae0>
CRITICAL:root:writing to <open file 'local-so.avro', mode 'wb' at 0x7fdb98674b70>
CRITICAL:root:writing to <open file 'local-nomanual.avro', mode 'wb' at 0x7fdb98674ae0>
CRITICAL:root:writing to <smart_open.smart_open_lib.S3OpenWrite object at 0x7fdb98624c90>
CRITICAL:root:writing to <smart_open.smart_open_lib.S3OpenWrite object at 0x7fdb985ef790>
ERROR:smart_open.smart_open_lib:encountered error while terminating multipart upload; attempting cancel
Traceback (most recent call last):
File "test_smartopenavro.py", line 129, in
write_avro_context_manager(foutd)
File "test_smartopenavro.py", line 68, in write_avro_context_manager
writer_contextManager.append(row)
File "/usr/anaconda/lib/python2.7/site-packages/avro/datafile.py", line 131, in exit
self.close()
File "/usr/anaconda/lib/python2.7/site-packages/avro/datafile.py", line 219, in close
self.flush()
File "/usr/anaconda/lib/python2.7/site-packages/avro/datafile.py", line 215, in flush
self.writer.flush()
AttributeError: 'S3OpenWrite' object has no attribute 'flush'
Traceback (most recent call last):
File "test_smartopenavro.py", line 129, in
write_avro_context_manager(foutd)
File "test_smartopenavro.py", line 68, in write_avro_context_manager
writer_contextManager.append(row)
File "/usr/anaconda/lib/python2.7/site-packages/avro/datafile.py", line 131, in exit
self.close()
File "/usr/anaconda/lib/python2.7/site-packages/avro/datafile.py", line 219, in close
self.flush()
File "/usr/anaconda/lib/python2.7/site-packages/avro/datafile.py", line 215, in flush
self.writer.flush()
AttributeError: 'S3OpenWrite' object has no attribute 'flush'
Since, i get the error, I also added another method which doesnt have manual close. that how i am trying to write now-
def write_avro_no_manual_close(foutd):
schema = avro.schema.parse(avroSchemaOut)
dictRes = data.to_dict(orient='records')
writer = avro.datafile.DataFileWriter(foutd, avro.io.DatumWriter(), schema)
for ll, row in enumerate(dictRes):
writer.append(row)
#writer.close()
def write_avro_context_manager(foutd):
schema = avro.schema.parse(avroSchemaOut)
dictRes = data.to_dict(orient='records')
with avro.datafile.DataFileWriter(foutd, avro.io.DatumWriter(), schema) as writer:
for ll, row in enumerate(dictRes):
writer.append(row)
@mock.patch('avro.datafile.DataFileWriter.generate_sync_marker', mock.Mock(return_value=b'0123456789abcdef'))
def write_avro_manual_close(foutd):
schema = avro.schema.parse(avroSchemaOut)
dictRes = data.to_dict(orient='records')
writer = avro.datafile.DataFileWriter(foutd, avro.io.DatumWriter(), schema)
for ll, row in enumerate(dictRes):
writer.append(row)
writer.close()
with open('local.avro', 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_context_manager(foutd)
with open('local-so.avro', 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_manual_close(foutd)
with open('local-nomanual.avro', 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_no_manual_close(foutd)
I clearly see that local.avro, local-so.avro are identical but not identical to local-nomanual.avro.
I am pasting the code i am using also for your reference-
import os
import os.path as P
import avro.io
import avro.datafile
import pandas as pn
import smart_open
import six
import subprocess
import warnings
import boto
from boto.compat import urlsplit, six
import boto.s3.connection
import logging
import json
Added by me
access_key_id=access_key_id
secret_access_key =secret_access_key
port=port
hostname=hostname
##########
with warnings.catch_warnings():
warnings.simplefilter('ignore')
import pandas as pn
logging.basicConfig(level=logging.ERROR)
if six.PY3:
assert False, 'this code only runs on Py2.7'
_S3_URL = 's3://bucket-for-testing/user/vinuthna'
assert _S3_URL is not None, 'please set the SO_S3_URL environment variable'
'''
_NUMROWS = os.environ.get('SO_NUMROWS')
if _NUMROWS is not None:
_NUMROWS = int(_NUMROWS)
'''
def gen_schema(data):
schema = {
'type': 'record', 'name': 'data', 'namespace': 'namespace',
'fields': [
{'name': field, 'type': ['null', 'string'], 'default': None}
for field in data.columns
]
}
return json.dumps(schema, indent=4)
def write_avro_nomanualclose(foutd):
schema = avro.schema.parse(avroSchemaOut)
dictRes = data.to_dict(orient='records')
writer_nomanual = avro.datafile.DataFileWriter(foutd, avro.io.DatumWriter(), schema)
for ll, row in enumerate(dictRes):
writer_nomanual.append(row)
#writer_manual.close()
def write_avro_context_manager(foutd):
schema = avro.schema.parse(avroSchemaOut)
dictRes = data.to_dict(orient='records')
with avro.datafile.DataFileWriter(foutd, avro.io.DatumWriter(), schema) as writer_contextManager:
for ll, row in enumerate(dictRes):
writer_contextManager.append(row)
def write_avro_manual_close(foutd):
schema = avro.schema.parse(avroSchemaOut)
dictRes = data.to_dict(orient='records')
writer_manual = avro.datafile.DataFileWriter(foutd, avro.io.DatumWriter(), schema)
for ll, row in enumerate(dictRes):
writer_manual.append(row)
writer_manual.close()
inputFilePath = P.join(_S3_URL,'test.csv')
splitInputDir = urlsplit(inputFilePath, allow_fragments=False)
#logging.info(">>> establishing connection for inputDir")
#establish connection manuelly for customized options
inConn = boto.connect_s3(
aws_access_key_id = access_key_id,
aws_secret_access_key = secret_access_key,
port=int(port),
host = hostname,
is_secure=False,
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
#get bucket
inbucket = inConn.get_bucket(splitInputDir.netloc)
read in the csv file
kr = inbucket.get_key(splitInputDir.path)
assert kr is not None, 'File not present'
with smart_open.smart_open(kr, 'r') as fin:
data = pn.read_csv(fin, header=1, error_bad_lines=False,dtype='str').fillna('NA')
num_csv_rows = len(data.index)
avroSchemaOut = gen_schema(data)
#No dataloss
with open('local.avro', 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_context_manager(foutd)
###No dataloss
with open('local-so.avro', 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_manual_close(foutd)
Data loss observed
with open('local-nomanual.avro', 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_nomanualclose(foutd)
##Below works
outputfilepath = _S3_URL + '/nomanual.avro'
splitoutdir = urlsplit(outputfilepath, allow_fragments=False)
kw = inbucket.get_key(splitoutdir.path,validate=False)
with smart_open.smart_open(kw, 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_nomanualclose(foutd)
This throws error
outputfilepath = _S3_URL + '/outcontext.avro'
splitoutdir = urlsplit(outputfilepath, allow_fragments=False)
kw = inbucket.get_key(splitoutdir.path,validate=False)
with smart_open.smart_open(kw, 'wb') as foutd:
logging.critical('writing to %r', foutd)
write_avro_context_manager(foutd)

Also, in your test code, try reading csv file from s3 because my test case is reading a csv from s3 -> converting to avro -> writing back avro to s3

Whoa... Please format your code next time.  Besides, most of that code is not relevant, because it demonstrates facts that we've already established (if you don't close the Avro writer, you'll lose some data).

Also, in your test code, try reading csv file from s3 because my test case is reading a csv from s3 -> converting to avro -> writing back avro to s3

Thank you for giving me details about your test case, I don't think this is relevant to the problem.
What version of avro are you using?

Sure,  I will format it. I am using version 2.0

Thanks. It says object is read only
Traceback (most recent call last):
File "test_smartopenavro.py", line 119, in
write_avro_nomanualclose(foutd)
File "test_smartopenavro.py", line 59, in write_avro_nomanualclose
foutd.flush = lambda: None
AttributeError: 'file' object attribute 'flush' is read-only

Sounds like you're overriding flush for the wrong object.

AttributeError: 'file' object attribute 'flush' is read-only

This means that flush already exists for that particular object.

Got it. Its  working now . Thanks a lot for the work around :)

Closed via https://github.com/RaRe-Technologies/smart_open/pull/212


Energy Calculations for Variable Speed Pumps

As reported by Marchi, A. and Simpson, A. (2013) and Georgeous et al. (2014), EPANET 2 does not properly take into account the affinity laws in the efficiency computation when the speed changes for variable speed pump (VSP) operations.
These researchers concluded that:

EPANET transposes correctly the head versus flow rate curve according to the affinity laws, but does not alter the efficiency curve of a pump that operates at a rotational speed different than the nominal speed. In this respect, energy consumption studies that imply variable speed pumps, performed with EPANET are unreliable.

I've checked and distinguished that the problem still exist in EPANET 2.1 release.! (In a test network with a VSP pump and demand pattern, daily energy consumptions and associated costs are the same for both EPANET 2.0 (of USEPA) and EPANET 2.1.
Solution
Marchi, A. and Simpson, A. (2013) proposed a correction in EPANET 2 source code for energy calculations for VSP operations.

Appendix I. How to Correct EPANET 2 Source Code to Compute the Efficiency of VSPs with the Affinity Laws To take into account the affinity laws for efficiency computation in the source code of EPANET 2 (Rossman 2000), it is sufficient to modify the hydraulic.c file of the source code and, in particular, the input to the function interp() called by getenergy(). The original function interp() computes the efficiency by interpolating the efficiency curve inserted in input. As it uses the flow computed for the specific time step at the reduced pump speed, q, and the efficiency curve at the nominal speed, it is necessary to modify one of these two inputs to retrieve the correct efficiency. The most efficient way to do this is converting the reduced speed flow q to the nominal speed flow, q4eff. In this way, the flow is rated to the nominal speed and can be used with the efficiency curve at the nominal pump speed. As in EPANET 2, the speed reduction is inserted as a speed rate (i.e., the nominal pump speed corresponds to the speed rate equal to 1), the reduced flow q is divided by the pump speed rate of the current time step. The modifications in the function getenergy() are shown in italics.

Appendix II. How to Modify EPANET 2 Source Code to Retrieve Pump Efficiency Directly To retrieve pump efficiency as an output of EPANET toolkit (Rossman 2000), three files of the source code need to be modified:toolkit.h, epanet.h, and epanet.c.As in the toolkit.h and epanet.h, different parameter codes are defined to refer to properties of the model components (e.g., the value 0 corresponds to the pipe diameter, the value 8 corresponds to the pipe flow), a new parameter code, 14, has been added to the list of link properties that can be retrieved using the function ENgetlinkvalue(). This function has been modified by consideringthe additional query for efficiency that is added at the end of the function. In the following, the modification of files and the function are highlighted in bold.

I hope this helps developers to correct the code for energy calculations in VSPs.

Thanks for the detailed writeup. Pull requests are welcome and encouraged!

I think this is literally a 1-line change. We have this correction in a development branch but neglected to incorporate here. If anyone wants to open the PR and get eternal fame, be my guest. Please PR into the dev branch.
https://github.com/samhatchett/EPANET/commit/716b440716e608c3f4eb3017723c7a77af6480b1

I have talked with Angela and once she gets into GH she will contribute this and other code she has.

#81 is merged so closing. Thanks all!


Fix translation service not being set by default

When first installing the app, if you set your API key then you'd still get the 'API key not set' warning. This was due to the translation service not being set.
This PR changes how this value is read from UserDefaults so that it can default to google, since that's what the picker did anyway so I've kept that functionality. I've also updated the picker so that a duplication of the @State isn't required.
First PR in this repo, hope it's alright. 👍🏼

@xiles Not sure if you get notified about this?


Using composite action

This pull request introduces the use of a composite action to streamline the workflow. The composite action is responsible for running the necessary steps to preview and publish content.
This is a replacement for #39

Although it helps organize workflows better, it also requires specifying the tokens multiple times in the files.
That is because it is not possible to use secrets in composite actions.

Maybe, It is better to save the secrets into env variables in the workflows.


enhance bids.report to list content of a dataset

Suggested by @rotemb9

creates a table with participants as rows and the different types of data (each run of each task, anat, fmap, etc) as columns, and then gives you the number of files per participant and data type.

similar table with file sizes so that if something is wrong with the size of the file it would be obvious to see

@rotemb9
This is currently work in progress on one of my branch: https://github.com/Remi-Gau/bids-matlab/tree/diagnostic_tools
Main function is this one: https://github.com/Remi-Gau/bids-matlab/blob/diagnostic_tools/%2Bbids/diagnostic.m

example output on ds000117
  BIDS = bids.layout('/home/remi/github/BIDS-matlab/tests/bids-examples/ds000117');
  diagnostic_table = bids.diagnostic(BIDS, 'output_path', pwd);
  diagnostic_table = bids.diagnostic(BIDS, 'split_by', {'task'}, 'output_path', pwd);

example output on ds000246
  BIDS = bids.layout('/home/remi/github/BIDS-matlab/tests/bids-examples/ds000246');
  diagnostic_table = bids.diagnostic(BIDS, 'output_path', pwd);
  diagnostic_table = bids.diagnostic(BIDS, 'split_by', {'task'}, 'output_path', pwd);

Thing to add is some basic checks on the events.tsv (plotting the number of conditions and trial/condition for each subject, task, run).
Also another visualization: a time course for each run of all the events divided by conditions (though this could get messy when the number or trial_types grows which can be the case with some HED tagging or for EEG with large numbers of trigger types).
https://github.com/cpp-lln-lab/CPP_SPM/pull/428#issuecomment-958989075

@all-contributors please add @rotemb9 for ideas

@rotemb9 this was on my to do for ages...


Fix SSH keys instead of 'SSH'

@snobu : Thanks for your contribution! The author, @heatherbshapiro, has been notified to review your proposed change.

#sign-off


proposal: io: add LimitWriter

In #51115, a proposal to add an error field to io.LimitedReader was accepted, but then the change was reverted when it turned out to have some tricky corner cases around the last byte.
As a new proposal, which may or may not supersede the change to io.LimitedReader, I suggest adding an io.LimitWriter function with approximately this implementation:
func LimitWriter(w io.Writer, limit int64, err error) io.Writer {
	return &limitedWriter{w, limit, err}
}

type limitedWriter struct {
	w io.Writer
	n int64
	e error
}

func (lw *limitedWriter) Write(p []byte) (int, error) {
	if lw.n < 1 {
		return 0, lw.e
	}
	if lw.n < int64(len(p)) {
		p = p[:lw.n]
	}
	n, err := lw.w.Write(p)
	lw.n -= int64(n)
	return n, err
}

Duplicate of #54111


[Flake]: Bigtable emulator crash during test.

The bigtable emulator crashed here:
https://source.cloud.google.com/results/invocations/4d0e30f1-a5e9-4a74-b6a7-13c97e55b510/targets/cloud-cpp%2Fgithub%2Fdocker%2Fclang-tidy/log
the relevant log lines are below.  I think we should upgrade to the latest and greatest google-cloud-sdk just on general principles. The line numbers do not match the source in:
https://github.com/googleapis/google-cloud-go/blob/master/bigtable/bttest/inmem.go
[ RUN      ]  table_admin_snippets drop-rows-by-prefix running
[    ERROR ]  table_admin_snippets drop-rows-by-prefix
================ [begin run_example.lGKnUA] ================
./table_admin_snippets drop-rows-by-prefix project-1553058091 in-1553058092 sample-table-for-admin-5715
Standard C++ exception raised: DropRowByPrefix(name=projects/project-1553058091/instances/in-1553058092/tables/sample-table-for-admin-5715) Socket closed
================ [end run_example.lGKnUA] ================
================ [begin emulator.log] ================
Cloud Bigtable emulator running on <IP_ADDRESS>:46379
panic: runtime error: index out of range
goroutine 64 [running]:
github.com/google/btree.(*node).iterate(0xc000ff62c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x100, 0xc00008dd18, 0x101)
	/usr/local/google/home/garyelliott/go/src/github.com/google/btree/btree.go:524 +0x88c
github.com/google/btree.(*node).iterate(0xc000ff6300, 0x1, 0x0, 0x0, 0x0, 0x0, 0xc000d90000, 0xc00008dd18, 0xc00067cb60)
	/usr/local/google/home/garyelliott/go/src/github.com/google/btree/btree.go:512 +0x5dd
github.com/google/btree.(*BTree).Ascend(0xc0011d4760, 0xc00008dd18)
	/usr/local/google/home/garyelliott/go/src/github.com/google/btree/btree.go:777 +0x5b
cloud.google.com/go/bigtable/bttest.(*table).gc(0xc000d906c0)
	/usr/local/google/home/garyelliott/go/src/cloud.google.com/go/bigtable/bttest/inmem.go:1188 +0x1e2
cloud.google.com/go/bigtable/bttest.(*server).gcloop(0xc0000ae400, 0xc000238840)
	/usr/local/google/home/garyelliott/go/src/cloud.google.com/go/bigtable/bttest/inmem.go:1097 +0x127
created by cloud.google.com/go/bigtable/bttest.(*server).needGC
	/usr/local/google/home/garyelliott/go/src/cloud.google.com/go/bigtable/bttest/inmem.go:1069 +0xa2
================ [end emulator.log] ================

Marking this blocked until we have more information.

Something like this happened again:
https://source.cloud.google.com/results/invocations/8c3451b2-30f3-43d2-81ed-3323769f1a74/targets/cloud-cpp%2Fgithub%2Fdocker%2Fshared/log
We are missing some logs to be able to troubleshoot.

Have not seen this in months. Closing.


Sak 41323

I hope it's okay to remove that second vector stuff.  It seemed kind of weird and unnecessary.  And why was it only on page 1 and 3?

It looks like they were alternating those List's based on the page which I am not sure why? been there since before 2006!


I've installed the theme on my blog and I have this problem.
The code bloc is not displayed correctly
Any hint or idea how to fix it ?
Thanks

Check whether your configuration file is configured according to the
# 我的配置
highlight:
  enable: false
  line_number: false
  auto_detect: true
  tab_replace: ''
  wrap: true
  hljs: true
prismjs:
  enable: false
  preprocess: true
  line_number: true
  tab_replace: ''

Yes I have the same configuration and I installed all the required package for Less compilation :/

是的，我有相同的配置，并且安装了 Less 编译所需的所有软件包：/

能否给你的仓库 或者MD我来尝试帮你解决这个问题

Yes I have the same configuration and I installed all the required package for Less compilation :/

Can you give me your code warehouse or your MD article file? I'll try to help you solve this problem.

Thanks for your help
This is the link for my repo
https://github.com/ahdbk/ahdbk.github.io
and this is an example of an md file
https://github.com/ahdbk/ahdbk.github.io/blob/master/source/_posts/tp-dotnet.md

I have looked at your code and found that the Quiet theme lacks the necessary javascript files. I have submitted a pr to you or you can clone the theme code again


Redeployed pod can't mount a csi-rbd-sc PVC - rbd image is still being used

Describe the bug
We have a Proxmox cluster with Ceph and k8s running there.
We are using the csi-rbd provider to deploy PVCs.
It is all seems to be working, a pod is started, pvc is created, a rbd image seems to be created, all is mounted and the app inside a pod is able to use it. Until the pod is deleted and recreated. From that moment, it won't start, as the PVC is not ready for the pod:
Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Normal   Scheduled               9m51s             default-scheduler        Successfully assigned monitoring/kube-prometheus-grafana-74bfb88f64-tctbf to k8s-dev-worker0
  Normal   SuccessfulAttachVolume  9m51s             attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-7e9588ad-19d1-4099-bd49-30fd2b9f0d42"
  Warning  FailedMount             17s (x9 over 9m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-7e9588ad-19d1-4099-bd49-30fd2b9f0d42" : rpc error: code = Internal desc = rbd image k8s-dev/csi-vol-dea80a4e-85b6-46e2-9d08-ffba10ef7825 is still being used

I think I was able to verify that the volume is really not mounted anywhere. I got a shell inside each ceph-csi-rbd-nodeplugin pod, on each worker node, and from the csi-rbdplugin container I was looking for the rbd device with something like rbd device ls | grep csi-vol-dea80a4e-85b6-46e2-9d08-ffba10ef7825. I double check directly on the worker node OS, by looking at all mount entries. Nothing found.
I also tried to mount the device manually. I picked a ceph-csi-rbd-nodeplugin container, prepared ceph and keyring config and I was able to map and mount: rbd -c /tmp/ceph.conf -k /tmp/ceph.keyring --id k8s-dev --pool k8s-dev map csi-vol-dea80a4e-85b6-46e2-9d08-ffba10ef7825, and then mount the block device on the host with like: mount /dev/rbd7 /mnt/. That worked. Now the rbd image is actually being used :)
But I can't find out why k8s things it can't be mounted. I need some help here please.
Is there a change that the storage.kubernetes.io/csiProvisionerIdentity could play any role in all of this? Does it change when the ceph-csi-rbd-provisioner is redeployed?
Because I noted that the csiProvisionerIdentity is different among persistent volumes and I did a couple of redeploys of the provisioner.
Environment details

Image/version of Ceph CSI driver :
Helm chart version : 3.10.1
Kernel version : 5.15.0-87-generic
Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
krbd or rbd-nbd) :
Kubernetes cluster version : 1.28.2
Ceph cluster version : 17.2.6

Steps to reproduce

create a deployment, backend with a rbd pvc
all works
delete the pod
pod get stucked in Init phase because of FailedMount - MountVolume.MountDevice failed for volume "pvc-7e9588ad-19d1-4099-bd49-30fd2b9f0d42" : rpc error: code = Internal desc = rbd image k8s-dev/csi-vol-dea80a4e-85b6-46e2-9d08-ffba10ef7825 is still being used

Nothing interesting from logs unfortunately, or at least I have not noticed.

funny, maybe the device really is mapped to the node :)
Because I can see there is a watcher on it:
user@hostname:~# rbd --id k8s-dev --pool k8s-dev status csi-vol-dea80a4e-85b6-46e2-9d08-ffba10ef7825
Watchers:
	watcher=<IP_ADDRESS>:0/3848453374 client.31966992 cookie=140516163400656

So, when I connect to the <IP_ADDRESS> server, I can't see it listed here:
rbd device ls | grep csi-vol-dea80a4e-85b6-46e2-9d08-ffba10ef7825

but when I try to unmap it anyway:
[root@k8s-dev-worker0 /]# rbd -c /tmp/ceph.conf -k /tmp/ceph.keyring --id k8s-dev --pool k8s-dev device unmap csi-vol-49be7950-9abe-4709-9560-b8813fdf1fe5
rbd: sysfs write failed
rbd: unmap failed: (16) Device or resource busy

So I'll keep looking, on the OS level probably. Something stale probably?

When you see the error rbd image k8s-dev/csi-vol-dea80a4e-85b6-46e2-9d08-ffba10ef7825 is still being used, there is probably something watching the RBD-image. On the Ceph cluster, you can check with rbd status $image, which should list the clients that have the image open. Ceph-CSI checks if there are any other processes using the image, if that is the case, it will not attach/mount the image to prevent potential data corruption.

Is it possible that the RBD image is mirrored to an other location? A process like rbd-mirror will have a watcher on the image too. There might be other processes that open the image with a watcher, but I am not familiar with them.


create-react-app build: Failed to minify the code from this file

Hi,
Using create-react-app (1.5.2) and stringify-object (3.3.0) i got the following error:
Creating an optimized production build...
Failed to compile.

Failed to minify the code from this file:

        ./node_modules/stringify-object/index.js:6

Read more here: http://bit.ly/2tRViJ9

Upgrade to Create React App v2. It supports automatic transpilation of dependencies, which will make this package just work.


include a simple fallback skin and container

Description of Problem
DNN includes a nice default theme package, which exposes many great features of DNN 8 f. This package consists of several skins and containers. As the default theme is used as well as fallback theme, if a skin or container fails to load, it cannot be deleted. as a result, all skin and container variants are always listed, when a user has the option to apply a theme to an individual page or a container to a page or module.
Proposed Solution Option 1
keep the current situation and ignore complaints.
Proposed Solution Option 2
no default skin - don't care about a site to break.
Proposed Solution Option 3
provide a minimalistic fallback skin (just with login, logo (, menu) and content pane) and container (title and contentPane) for emergency cases. they should be named default.skin and default.container and not be listed in dropdowns.
Approved Solution
Please see the following comment. https://github.com/dnnsoftware/Dnn.Platform/issues/2565#issuecomment-479171171
Affected version

[x] 9.3.0
[x] 9.2.2
[x] 9.2.1
[x] 9.2
[x] 9.1.1
[x] 9.1
[x] 9.0

Option 3 is a great idea

Option 3 for sure, however before we start to add additional skins into the system we should address the poor performance related to having multiple skins available. I’ll create an RFC which highlights the issues with the skin packages.

At the @dnnsoftware/approvers meeting today we all like the idea of creating a failback skin. We are switching this from an RFC to a help wanted request.
We are looking for a new failure skin to be submitted, plus reproducible steps to trigger a failure in the main skin, and in addition a failure in the failback skin.
This way we can test for no failures, a failure in the  main skin which triggers the failback skin to render successfully, and a failure in the failback skin which would then trigger the no skin failure.

I believe in DNN 8, there used to be some entries in the hostsettings:

DefaultPortalContainer
DefaultPortalSkin
DefaultAdminContainer
DefaultAdminSkin
Which kind of worked as a fallback.

And then ofcourse if all failed there was indeed:
\Portals_default\Skins_default\No Skin.ascx
And
\Portals_default\Containers_default\No Container.ascx
Which are about as minimal as you can get....
And because of the '_' in the folder name, they could not be seen from within DNN I believe.
So perhaps they can still be used. Though they could do with some styling so they look at least a bit better...

I kinda agree with @trouble2 I think the fallback is already in place, I think we could just style that a bit but stay very minimal.

IMO this is still an issue

I would prefer adding a skin setting "show in PB", that would fix this and could be useful in other scenarios.

took me a while to think about this.  Option 3 seems to be favorite here and I like to go with the flow but was trying to play devils advocate.
I am trying to fully understand the issue and the proposed solution given all the recent developments.   What solution would be acceptable so I can think if it is something I can work on if no one else can get around to this anytime soon.  Seems like minor changes potentially, however I may not be fully aware of what all is being asked to perform to be satisfied for a solution.
From what I understand is you can or cannot uninstall Xcillion or lets say the theme was modified and fails to load.  This theme is the only theme on your installation and you need to get it up and running again.  What happens then?
Am I correct so far?
If so then the current fallback theme files for DNN would be
\Portals_default\Skins_default\No Skin.ascx
and
\Portals_default\Containers_default\No Container.ascx
So are these the files that need Tuned?  And if so how, what is missing or needs adjusted to make a solution work?

<%@ Control Language="C#" CodeBehind="~/DesktopModules/Skins/skin.cs" AutoEventWireup="false" Inherits="DotNetNuke.UI.Skins.Skin" %>
<div runat="server" id="ContentPane" />

Very simple skin here so anything that is needed?  I am not sure the container needs tuned as it probably is just fine.

At the @dnnsoftware/approvers meeting today we all like the idea of creating a failback skin. We are switching this from an RFC to a help wanted request.
We are looking for a new failure skin to be submitted, plus reproducible steps to trigger a failure in the main skin, and in addition a failure in the failback skin.
This way we can test for no failures, a failure in the main skin which triggers the failback skin to render successfully, and a failure in the failback skin which would then trigger the no skin failure.

So when a skin fails it needs to catch the error and then load a fallback skin which would be in these two files mentioned above?
And I like asking for more details on what are any known triggers for the failures so they can be tested.

A failure in the failback skin should go to the two files above am I right?
And the primary failback skin possibly should be able to get set by an admin but use a default one or  a skin that no one sees?
I think removing all skins to see what the current failback might give some results to build on.  Other than that something would have to be done to catch the error of the skin and then send it to the next skin, then to the next skin until the page finally loads putting an error in the admin logs.

Not sure if everybody realizes this but the "error skin" is set in dotnetnuke.config.
When you change this to a another skin you can uninstall xcillion
  <skinningdefaults>
      <skininfo folder="/Xcillion/" default="Inner.ascx" admindefault="Admin.ascx" />
      <containerinfo folder="/Xcillion/" default="Title_h2.ascx" admindefault="Title_h2.ascx" />
  </skinningdefaults>

IMO users should never edit the default skin, they should always make a copy and edit that, as the default skin could be overwritten on upgrade and all changes lost.
So IMO it's a minor issue as there are other issues in this area that are way more important TBH.

Editing the default skin in place is a big no no for sure, very bad practice, no core files should be edited by users since they can get ovewritten by upgrades.
I am not 100% sure about this, but I think dotnetnuke.config is only used upon installation, what allows uninstalling Xcillion is to not have it mapped to any page in the database. I might be wrong though.

OK I'll explain how it works: https://www.youtube.com/watch?v=hwyLRXM28ds :-)

As a reference, you can trigger a skin error by including this:
<dnn:error runat="server"  />
(the word error can be anything else as long ad there's no reference at the top of the page)

Thanks for double-checking that @Timo-Breumelhof so yeah, I was wrong and this applies not only during installation. Thanks.

Is there still a need for this?

default skin info is saved in DotNetNuke.config file under the site root folder.

IMO yes

IMO yes

still an issue


Watch fn

fix for: https://groups.google.com/forum/#!topic/angular/FCh9C1BHmpg

I'm not sure how i can use it.
Currenlty i'm using AngularJS v0.10.5 and i add/replace  isScope() and equals() from
https://github.com/IgorMinar/angular.js/blob/9aff6600f74bdf56035fc329b87c3adbaf1ac139/src/Angular.js
And this solves problem. But i'm aware that i may miss something that may cause some problems in future.

@SET
Bite the bullet and upgrade to v0.10.6 :-)
Then build after Git pull to local.

@SET you can't just take a random file from one angular version and mix it with files from another angular version, build the js and expect it to work :-)
Taking just the two functions you mentioned sounds like a good approach if you don't want to upgrade to 0.10.6 and beyond right now.

LGTM

[angular.js] IgorMinar pushed 3 new commits to master: http://git.io/BtXmOw


⚠ Bofrost.es has degraded performance

In 1063d56, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1522 ms

Resolved: Bofrost.es performance has improved in d8fff61 after 6 minutes.


Export with -ref flag occasionally produces references with blank names

Running terraform-provider-dynatrace -export -id -ref dynatrace_aws_service occasionally produces TF files with blank name references.
Error Message:
Error: Invalid attribute name

  on modules/aws_service/ELB_builtin.aws_service.tf line 4, in resource "dynatrace_aws_service" "ELB_builtin":
   4:   credentials_id = "${var.dynatrace_aws_credentials..id}"

An attribute name is required after a dot.

No longer reproducible after recent fixes to the export utility


[WIP] feature: finish upgrade interface

Signed-off-by: Michael Wan<EMAIL_ADDRESS>Ⅰ. Describe what this PR did
finish upgrade interface daemon part
root@osboxes:wanziren -> pouch upgrade --name test1 registry.hub.docker.com/library/busybox:latest
test1

Ⅱ. Does this pull request fix one issue?
Ⅲ. Describe how you did it
Ⅳ. Describe how to verify it

Create a new container

root@osboxes:pouch (zr/finish-upgrade) -> pouch inspect test2
{
    "Args": null,
    "Config": {
        "Cmd": [
            "sh"
        ],
        "Entrypoint": null,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Image": "registry.hub.docker.com/library/busybox:latest",
        "OnBuild": null,
        "Shell": null
    },
    "Created": "2018-03-20T02:19:05.418084614Z",
    "HostConfig": {
        "NetworkMode": "bridge",
        "RestartPolicy": {
            "Name": "no"
        },
        "Runtime": "runc",
        "BlkioDeviceReadBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceWriteIOps": null,
        "BlkioWeightDevice": null,
        "CgroupParent": "default",
        "DeviceCgroupRules": null,
        "Devices": [],
        "Memory": 20971520,
        "MemoryExtra": 0,
        "MemorySwappiness": -1,
        "MemoryWmarkRatio": 0,
        "Ulimits": null
    },
    "Id": "4c58d27f58d38776dda31c01c897bbf554c802a9b80ae4dc20be1337f8a969f2",
    "Image": "registry.hub.docker.com/library/busybox:latest",
    "Mounts": null,
    "Name": "test2",
    "NetworkSettings": {
        "Networks": {
            "bridge": {
                "Aliases": null,
                "EndpointID": "e3845861204350e6ec1ac2fbfaece7d3fe384ed0f5c01b41b53478d89b70a131",
                "Gateway": "<IP_ADDRESS>",
                "IPAddress": "<IP_ADDRESS>/24",
                "IPPrefixLen": 24,
                "Links": null,
                "MacAddress": "02:42:ac:11:00:02",
                "NetworkID": "59a8a8a365171ea1862f11534be913a81a6b4fc39f516b3121fb71c52b3dc2f8"
            }
        },
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "State": {
        "Pid": 4895,
        "StartedAt": "2018-03-20T02:19:05.780853724Z",
        "Status": "running"
    }
}

Update container's image

root@osboxes:pouch (zr/finish-upgrade) -> pouch upgrade --name test2 registry.hub.docker.com/library/ubuntu:14.04
test2

Check the container's status

root@osboxes:pouch (zr/finish-upgrade) -> pouch ps
Name    ID       Status          Created          Image                                                Runtime
test2   4c58d2   Up 2 minutes    15 minutes ago   registry.hub.docker.com/library/ubuntu:14.04         runc

Inspect the container's meta info

root@osboxes:pouch (zr/finish-upgrade) -> pouch inspect test2
{
    "Args": null,
    "Config": {
        "Cmd": [
            "sh"
        ],
        "Entrypoint": null,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Image": "registry.hub.docker.com/library/ubuntu:14.04",
        "OnBuild": null,
        "Shell": null
    },
    "Created": "2018-03-20T02:19:05.418084614Z",
    "HostConfig": {
        "NetworkMode": "bridge",
        "RestartPolicy": {
            "Name": "no"
        },
        "Runtime": "runc",
        "BlkioDeviceReadBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceWriteIOps": null,
        "BlkioWeightDevice": null,
        "CgroupParent": "default",
        "DeviceCgroupRules": null,
        "Devices": [],
        "Memory": 20971520,
        "MemoryExtra": 0,
        "MemorySwappiness": -1,
        "MemoryWmarkRatio": 0,
        "Ulimits": null
    },
    "Id": "4c58d27f58d38776dda31c01c897bbf554c802a9b80ae4dc20be1337f8a969f2",
    "Image": "registry.hub.docker.com/library/ubuntu:14.04",
    "Mounts": null,
    "Name": "test2",
    "NetworkSettings": {
        "Networks": {
            "bridge": {
                "Aliases": null,
                "EndpointID": "e3845861204350e6ec1ac2fbfaece7d3fe384ed0f5c01b41b53478d89b70a131",
                "Gateway": "<IP_ADDRESS>",
                "IPAddress": "<IP_ADDRESS>/24",
                "IPPrefixLen": 24,
                "Links": null,
                "MacAddress": "02:42:ac:11:00:02",
                "NetworkID": "59a8a8a365171ea1862f11534be913a81a6b4fc39f516b3121fb71c52b3dc2f8"
            }
        },
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "State": {
        "Pid": 8251,
        "StartedAt": "2018-03-20T02:32:43.874327747Z",
        "Status": "running"
    }
}

Ⅴ. Special notes for reviews

Codecov Report

Merging #923 into master will decrease coverage by 0.14%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #923      +/-   ##
==========================================
- Coverage   13.59%   13.45%   -0.15%
==========================================
  Files         121      121
  Lines        7706     7790      +84
==========================================
  Hits         1048     1048
- Misses       6568     6652      +84
  Partials       90       90

Impacted Files
Coverage Δ

pkg/utils/utils.go
82.08% <ø> (ø)
:arrow_up:

cli/upgrade.go
0% <0%> (ø)
:arrow_up:

client/container.go
0% <0%> (ø)
:arrow_up:

daemon/mgr/container.go
2.21% <0%> (-0.2%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2db13e...2f087b2. Read the comment docs.

Thanks a lot for the work you made. I am wondering adding some integration test will guarantee this  much better. @HusterWan 💟

CI fails:
FAIL: /go/src/github.com/alibaba/pouch/test/cli_upgrade_test.go:58: PouchUpgradeSuite.TestPouchUpgradeNoChange
/go/src/github.com/alibaba/pouch/test/cli_upgrade_test.go:64:
    c.Assert(res.Error, check.NotNil)
... value = nil

@HusterWan

LGTM
And I think we need to add a document to tell users how to take advantages of this API. WDYT? @HusterWan

And I think we need to add a document to tell users how to take advantages of this API. WDYT? @HusterWan

Yes, i will add the doc after i have a tiime, but i will create a issue to reminder me.


device: esp32: Convert clock control to use DEVICE_DT_GET

Replace device_get_binding with DEVICE_DT_GET for getting access
to the clock controller device.

blocked on #32188


Improve German translation

Thanks for creating this project!
When first installing this plugin and configuring my bridge, some phrases seemed a bit odd to me. In this PR, you’ll find some suggestion on how I’d reword them.

Daaaanke!


Change version check : android - 48 and ios 60

:tada: This PR is included in version 1.35.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Include function

Include function
Description

ok so i am trying to include the scaldeone api script
but the script can not be loaded out of the server so i can only load it  from
https://cdn.scaledrone.com/scaledrone.min.js
but the problem is that i can only include scripts from the program.. i was wondering if you could do something like

include("https://cdn.scaledrone.com/scaledrone.min.js","url")

Version

Platform

Web Version [X]
Windows [X]
Linux [X]

Implemented this feature in the last web version of nunuStudio.


Community Guidelines for contribution

As per JOSS guidelines, there needs to be community guidelines:
" third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support"
This could be added to the wiki or the README

Community guidelines were added to the README file.


Resources on Temporal Normalization

Description
Hey all!
A colleague and I have been utilizing the neuralforecast. One of the important features that works well with our data is the invariant scaling. We need to be able to justify its use to our risk validator. We will be doing a deep dive but are also wondering if there are already resources that we can read to understand the mindset and philosophy of the scaler.
Link
https://nixtla.github.io/neuralforecast/common.scalers.html#invariant_statistics

I may have answered my own question:

https://journalspress.com/LJRS_Volume19/600_Data-Normalization-using-Median-&-Median-Absolute-Deviation-(MMAD)-based-Z-Score-for-Robust-Predictions-vs-Min–Max-Normalization.pdf
https://pro.arcgis.com/en/pro-app/latest/help/analysis/geostatistical-analyst/box-cox-arcsine-and-log-transformations.htm

👍

@rydeveraumn: we can include your summary or document in the documentation. It would be a great addition.

Hey @mergenthaler yes I would love too - I was thinking of making a notebook with these resources but before I do let me know if that is something you believe the community will be interested in.

That sounds great!


[WebToolsE2E][Aspire] Installing Aspire 8.0.1 throws an exception: Cannot access a disposed object System.Collections.Concurrent.BlockingCollection.

REPRO STEPS

Clean machine: Win11 x64 23h2 ENU
Install SDK 8.0.302
Install Aspire 8.0.1 using commands 'dotnet workload update' and 'dotnet workload install aspire'

ACTUAL
The aspire workload installed successfully but encountered an error
Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Collections.Concurrent.BlockingCollection`1[[System.String, System.Private.CoreLib, Version=<IP_ADDRESS>,
Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]'.
at Microsoft.DotNet.Installer.Windows.TimestampedFileLogger.WriteMessage(String message)
at Microsoft.DotNet.Installer.Windows.TimestampedFileLogger.ProcessLogRequests(Object logPipeName)

This issue rarely occurs, reporting it in case we miss something.

Adding @marcpopMSFT @joperezr for awareness.

cc @dsplaisted @Forgind to take a look. This is 8.0.302 which is the hotfix June release we did for Razor I believe. Did this repro with 8.0.300 (as we didn't make any changes in the SDK between those builds)? Does it repro with 8.0.302?

Looks like a race condition or similar in TimestampedFileLogger between disposing the object and writing out the messages.  Since it's a timing issue, it could start occurring without any specific code changes in that area.  Probably we need to add a lock in WriteMessage and Dispose.  @joeloff


Initial proposal for /allow endpoint

This creates an /allow endpoint which does the actual permission
check.
It takes three mandatory query parameters:

tenant: The tenant URN
resource: The resource URN
action: The action identifier tag

It also adds a simple OpenAPI v3 spec document.
The intent is to have an easy, general and fairly opinionated endpoint
to do permission checks on. This can be taken programmatically without
much logic in... say... an API Gateway, to do the needed checks without
adding much logic other than gathering the mandatory parameters.
Signed-off-by: Juan Antonio Osorio<EMAIL_ADDRESS>

@fishnix thansk for the feedback, I think we can address that by making the resource ID optional.


Parallelize samtools_sort

Brings back #38.

I found sorting with @ didn't really speed up anything, but that was a few versions back. Thank you for the PR!


Portuguese translation to V3

Portuguese adaptation of course from spacy 2 to spacy 3.
Used the changes made in the english version of the course as a baseline of comparison, from PR #104

I have also adapted all exercises that were using the english pipeline to use the portuguese pipeline.
I have tested all exercises and they are working fine.
Except for:
exercise 02.13: it seams that the tokenizer is not working properly
exercise 03.1: pipeline did not extract PERSON and GPE

@ines , as soon as you have updated the course in the platform, I will do a final review :)

@Cristianasp Thanks 🙏 I've updated the PR and tests but it seems like there are still a couple of test failures related to the exercises where the expected output doesn't match the test: https://dev.azure.com/explosion-ai/Public/_build/results?buildId=16026&view=logs&j=f1d52934-b974-5ece-337e-e658a4411382&t=25745add-d8f1-5bdf-c7de-2e51e93603d8&l=1164
Could you take a look at them? Maybe the tests just need to be updated here or we need to pick different examples 🙂

Here it goes ! All portuguese exercises reviewed and tested :-) It seams that everything is ok now. 🙏

Awesome, thank you! 🙏 Will go ahead and merge this now and update the Binder image and then you can test it one more time once it's live. And if it all looks good, we'll go ahead and promote it to the community 😊


Accomodate AIX install, which has different syntax

for #1803

Okay, it is not recognizing the $OSNAME, so I will have to research what the solution should be.  Will update when I figure it out.

Probably just a misplaced parenthesis in my commit - should be "($(OSNAME),AIX)" not "(($OSNAME),AIX)" sorry...

Any advice on why these files are not "found"?  It is there:
$ sudo make install
make -j 2 -f Makefile.install install
make[1]: Entering directory '/medstat/advantage/scratch/focal_point/download/OpenBLAS'
Generating openblas_config.h in /opt/OpenBLAS/include
Generating f77blas.h in /opt/OpenBLAS/include
Generating cblas.h in /opt/OpenBLAS/include
Copying LAPACKE header files to /opt/OpenBLAS/include
find: bad status-- /opt/OpenBLAS/include/lapacke.h
install: File lapacke.h was not found.
make[1]: [Makefile.install:46: install] Error 2 (ignored)
find: bad status-- /opt/OpenBLAS/include/lapacke_config.h
install: File lapacke_config.h was not found.
make[1]: [Makefile.install:47: install] Error 2 (ignored)
find: bad status-- /opt/OpenBLAS/include/lapacke_mangling.h
install: File lapacke_mangling_with_flags.h.in was not found.
make[1]: [Makefile.install:48: install] Error 2 (ignored)
find: bad status-- /opt/OpenBLAS/include/lapacke_utils.h
install: File lapacke_utils.h was not found.
make[1]: [Makefile.install:49: install] Error 2 (ignored)
Copying the static library to /opt/OpenBLAS/lib
install: File libopenblas_power5p-r0.3.3.a was not found.
make[1]: *** [Makefile.install:51: install] Error 2
make[1]: Leaving directory '/medstat/advantage/scratch/focal_point/download/OpenBLAS'
make: *** [Makefile:336: install] Error 2
Here's the file status:
$ find . -name lapacke.h
./lapack-netlib/LAPACKE/include/lapacke.h
$ find . -name lapacke_config.h
./lapack-netlib/LAPACKE/include/lapacke_config.h
$ find . -name lapacke_mangling.h
./lapack-netlib/LAPACKE/include/lapacke_mangling.h
$ find . -name lapacke_utils.h
./lapack-netlib/LAPACKE/include/lapacke_utils.h
$ find . -name libopenblas_power5p-r0.3.3.a
./libopenblas_power5p-r0.3.3.a

The only somewhat sane search result that I found (from comp.unix.aix ca. 10 years ago) suggests that the AIX install expects the destination argument to be the target directory rather than the full name of the target file. (So no on-the-fly renaming with this tool it seems.) Perhaps we should change this again to use the installbsd variant (is this available by default on AIX ?)

I will have to ask around, but it's there for me:
$ installbsd
usage: install [-cs] [-g group] [-m mode] [-o owner] file1 file2;
or file1 ... fileN directory


[FEATURE] Make use of ClientId when using Connect-PnPOnline -Connection $ContextWithClientId

Is your feature request related to a problem? Please describe.
I have scripts that use the following variables
$AdminConn = Connect-PnPOnline -Url $adminurl -Interactive -ReturnConnection
$SiteConn = Connect-PnPOnline -Connection $AdminConn -Url $site -Interactive -ReturnConnection
I'm changing my scripts to make use of a custom App Reg, so it would be handy to have the variables like this
$AdminConn = Connect-PnPOnline -Url $adminurl -ClientId $clientid -Interactive -ReturnConnection
$SiteConn = Connect-PnPOnline -Connection $AdminConn -Url $site -Interactive -ReturnConnection
But this throws "WARNING: Connecting with -Interactive uses the PnP Management Shell....."
This shows me that the ClientId in the $AdminConn is being ignored when using -Connection $AdminConn
Describe the solution you'd like
Make use of the ClientId when using -Connection so -ClientId doesn't need to be used on subsequent connections
Describe alternatives you've considered
The workaround is using the following variables
$AdminConn = Connect-PnPOnline -Url $url -ClientId $clientid -Interactive -ReturnConnection
$SiteConn = Connect-PnPOnline -Connection $AdminConn -Url $site -ClientId $clientid -Interactive -ReturnConnection
Additional context
None really, It's not a huge pain to add -ClientId to the various Connections made in my scripts, it just seems unnecessary (Kinda like needing -Interactive when the -Connection uses a context that was already -Interactive...)

@Grendel04 - Can you please try with the latest nightly and let us know if this is still the case ? We have improved this part of the code, should help.

@Grendel04 - Can you please try with the latest nightly and let us know if this is still the case ? We have improved this part of the code, should help.

No change with the nightly build. I still need -ClientId to avoid the Warning and the additional Interactive popup.

@Grendel04 - is a valid bug indeed. To save the trouble of changing things in your scripts, you can also specify an environment variable ENTRAID_APP_ID with the value of your AppId/ClientId.
Like this :
$Env:ENTRAID_APP_ID = "<Client-Id>"

We have modified the code to pick up value from the variable.
https://pnp.github.io/powershell/articles/defaultclientid.html
Will try and fix the bug for sure, keeping this open.

Not sure if this could be called a bug. It has never been built to work this way. I have created a PR https://github.com/pnp/powershell/pull/4425 which mitigates what is being asked for here. Not sure if it makes things better or only less intuitive though. You could also simply pass in -ClientId $AdminConn.ClientId on your second connect, right?

I've implemented both methods in various scripts.
Thank you both!


discover multiple printers

i am using this code in my application
   try {
      EscPosPrinter.discover().then((printers) => {
        console.log("PRINTERS!", printers);
        setPrinters(printers);
      });
    } catch (error) {
console.log(error.message)
}

but it is showing only 1 printer which i found earlier.
Now I turned on 3 more epson printers but it is still showing the older one.
If i turn off the older one, then it is returning empty array [ ] on above discover method, even other devices are powered on.

@Rehankhalil462 please try this
https://github.com/tr3v3r/react-native-esc-pos-printer/issues/54#issuecomment-1110858740
and the issues is duplicate
https://github.com/tr3v3r/react-native-esc-pos-printer/issues/40

@tr3v3r  do i need to patch this mentioned code into the library on my side? or i can directly use this in my RN application?

Fixed in v.3.0.0
https://github.com/tr3v3r/react-native-esc-pos-printer/releases/tag/vv3.0.0


[Idea] Dynamic facade generation to overcome storage limitation

Currently, ScalablyTyped publishes its type facades to Bintray.
Unfortunately, Bintray free plan has 10GB storage limitation, which is small for ScalablyTyped's big ambition to provide type facade for all JS ecosystems !!
So cross-publishing for Scala versions and Scala.js versions is very unlikely, unless storage limitation is addressed, as mentioned in #7
(I could not found limitation about Sonatype OSS Repository Hosting. You may give it a try?)
My idea: ScalablyTyped become a sbt plugin to generate facades dynamically as managed source on demand.
If it is possible, you do not need to worry about storage limitation any more.
Something like this.
addSbtPlugin("org.scalablytyped" % "sbt-scalablytyped" % "xxxxxx")

val myAwesomeProject = project(".")
   .enablePlugin(ScalablyTypedPlugin)
   .settings(
      generatedTypeFacades := Seq(
        "d3" -> "5.5.0", // Generated on compile-time
      ),
      npmDependencies  in Compile := Seq(
        "d3" -> "5.5.0",
      )
   )

...Since I do not know much about managed source in sbt, this may be a stupid idea.

Event I do not understand this fully, but this question has bothered me too. Why is ScalablyTyped not generating facades dynamically? Thanks for bring this out.

Hey there. Sorry, I didn't see notifications for this somehow.
Thanks for bringing this up, it is not a bad idea, and indeed a direction the project might take in the future. It is also an idea I have considered and decided not to pursue (for now). The reasons why I chose the direction of publishing are the following:
Performance
People care about how long their build takes, and compiling ScalablyTyped is not fast. Compiling just the stdLib alone takes 60-200 seconds, and that's a prerequisite for almost all libraries. UI toolkits like material-ui takes 500 seconds, and the really big libraries (aws-sdk comes to mind) takes 40 min.
There is also the time taken to convert, and to download resources... My converter has not been written with performance in mind.
Sharing code on top of typings
If all typings are locally generated source code, we have no way of sharing code to make them more usable. I'm talking about the facades in particular, but it could be any external library.
I think this part of ScalablyTyped is very important for success, and we would need to come up with something to replace it.

For now I hope to find a solution to the storage problem (I have some ideas), and I'll be pretty happy with that. Once I get around to open sourcing the converter itself anyone can give this a shot.

How much storage and transfer is (currently) roughly necessary?

@an-tex one full build for one specific combination of scala and scala.js versions weighs a bit less than two gigabytes. We currently host around five releases of a given library before it's deleted


a couple errors when building

I encountered a couple error messages when building, not sure if they are expected or not.
The output of sudo /vagrant/bin/capture-stack foo looked good, but then at the end:
-----> Cleaning up. Logs at /tmp/log/capture-stack.log
       umount: /tmp/cedar64-foo: not mounted

The output of sudo /vagrant/bin/install-stack /tmp/cedar64-foo.img.gz looked good, but then at the end:
-----> Mounting image /mnt/stacks/cedar64-foo
/mnt/stacks/cedar64-foo.img: No such file or directory

The commands are relying on each other. What are you executing exactly?

the recipe from the readme. i'll close this, as #30 suggests that this project isn't for me anyway :)


Suppported languages

Is it possible to support non latin languages ?

You can see what it is possible to do directly in Google lib:
https://pub.dev/packages/google_mlkit_text_recognition#usage


Trezor for Desktop Wallet on Mac not working

I have the latest software, and latest firmware on Trezor Model T.
After clicking "Link the Device" it shows a modal that says "Check your Trezor for address".
Afterwards, I click on the "Device" button on the Trezor and both screens go blank (TezBox/Trezor).
Same exact situation happens when I click the "Host" button.

Host button? I'm not familiar with that, could you explain?

Yeah for sure. If you have a Trezor Model T and you click "Link Tezbox" on the TezBox software it comes up with a binary option on the Model T screen that asks whether you want to enter password on "Host" (meaning TezBox) or "Device"

Oh ok, right I must have that disabled on my device. For now that's not working - I believe that would require some additional work. I will investigate that and try to push out a patch asap

Cool, sounds good.

Good news, passphrase support has been completed on our development version and has passed testing on Windows. Will be running through testing on MacOS, and hope to post a release today/tomorrow. Thanks!

you're very welcome, awesome job! thanks for the update

Latest release should fix this: https://github.com/tezbox/desktop-wallet/releases/tag/2.0.1


Full movement in space while magboots toggled on bug

Description
While in space, player has full movement and ability to walk around while having magboots toggled on.
Reproduction
Equip Magboots, Go into space, Turn on magboots. Walk around in space.
Screenshots
None
Additional context
Full movement in space bug!

dupe and the fix is already up